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EXECUTIVE  SUMMARY 

This  report  describes  in  detail  the  procedures,  methods,  and  resources  Parsons  used  to  complete 
the  demonstration  project  at  the  Massachusetts  Military  Reservation  for  Environmental  Security 
Technology  Certification  Program  (ESTCP)  Munitions  Response  (MR)-201104  (Evaluation  and 
Discrimination  Technologies  and  Classification  Results)  and  ESTCP  MR-201157 
(Demonstration  of  MetalMapper  Static  Data  Acquisition  and  Data  Analysis).  The  2011-2012 
ESTCP  Unexploded  Ordnance  (UXO)  Classification  Study,  Massachusetts  Military  Reservation 
(MMR),  was  conducted  with  three  primary  objectives: 

•  Test  and  validate  detection  and  discrimination  capabilities  of  currently  available  and 
emerging  advanced  electromagnetic  induction  sensors  developed  specifically  for 
discrimination  on  real  sites  under  operational  conditions. 

•  Investigate  in  cooperation  with  regulators  and  program  managers  how  classification 
technologies  can  be  implemented  in  munitions  and  explosives  of  concern  (MEC)  cleanup 
operations. 

•  Identify  and  remove  large  target  items  as  they  pose  the  greatest  potential  for  groundwater 
contamination  from  the  energetic  filler  in  the  UXO. 

Parsons  had  two  separate  teams  working  on  the  project  under  two  different  ESTCP  project 
numbers,  ESTCP  201104  and  ESTCP  201157.  One  team  was  responsible  for  site  setup,  the 
placement  of  30  seed  items  for  use  in  measuring  the  capabilities  of  the  MetalMapper  advanced 
electromagnetic  induction  (EMI)  sensor  tested  during  the  project,  and  the  intrusive  investigation 
of  the  2,287  targets  (including  seed  items)  selected  for  additional  investigation  with  the  advanced 
sensors.  The  second  team  was  responsible  for  the  cued  survey  of  2,287  targets  with  the 
MetalMapper.  These  targets  were  selected  from  EM61-MK2  data  collected  by  the  National 
Guard  Bureau  during  their  Impact  Area  Groundwater  Study  Program  (lAGWSP),  a  separate 
effort  from  the  ESTCP  demonstration.  EM6I-MK2  targets  were  selected  in  locations  designed 
to  test  the  capabilities  of  the  MetalMapper  with  regard  to  a  much  higher  anomaly  density  than 
had  been  present  on  previous  demonstration  sites.  The  field  MetalMapper  field  collection  effort 
took  place  over  3  weeks,  and  the  average  production  rate  was  207  points  per  day. 

The  MetalMapper  is  an  advanced  EMI  system  developed  by  Geometries,  Inc.,  with  support  from 
the  ESTCP.  It  has  three  mutually  orthogonal  transmit  loops  in  the  Z,  Y,  and  X  directions  and 
contains  seven  triaxial  receiver  antennas  inside  the  Z  (bottom)  loop,  allowing  21  independent 
measurements  of  the  transient  secondary  magnetic  field.  Data  were  collected  statically,  such  that 
one  data  point  was  collected  for  each  target  selected  for  investigation.  The  collected  data  were 
inverted  and  analyzed  using  the  UX-Analyze  add-on  to  Geosoft’s  Oasis  montaj  software.  Once 
analysis  was  complete,  a  theoretical  ranked  dig  list  (theoretical  because  all  targets  were 
intrusively  investigated  regardless  of  demonstrators’  stop-dig  points)  was  submitted  for  scoring 
by  the  Institute  for  Defense  Analyses  (IDA). 

Dig  list  scoring  was  based  on  the  number  of  targets  of  interest  (TOI)  correctly  identified  as  items 
that  should  be  dug  and  the  number  of  non-TOI  or  clutter  items  that  were  correctly  classified  as 
items  that  did  not  need  to  be  intrusively  investigated.  The  intrusive  investigation  was  split  into 
two  phases,  I  and  II,  and  a  separate  ranked  dig  list  was  submitted  for  each  phase.  The  dig  lists 
submitted  were  scored  against  the  ground  truth  set  compiled  following  the  intrusive 
investigations.  Comparison  of  the  Phase  I  dig  list  with  the  Phase  I  ground  truth  set  resulted  in 
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the  correct  identification  of  approximately  97%  of  the  TOI  recovered  in  the  intrusive 
investigation  (5  of  148  TOI  incorrectly  classified  as  clutter)  and  a  reduction  in  the  amount  of 
clutter  that  would  have  been  dug  by  approximately  78%.  The  missed  TOI  included  three  items 
considered  too  small  to  be  of  concern  at  MMR  (only  small  amounts  of  energetic  filler)  and  one 
item  with  questionable  location  compared  to  the  MetalMapper  collection  location.  The  one  item 
of  concern  identified  as  a  false  negative  was  a  60-millimeter  (mm)  mortar  illumination  round. 
This  single  round  was  not  deemed  concerning  enough  to  add  to  the  library  prior  to  the 
compilation  of  the  Phase  II  list.  Comparison  of  the  Phase  II  dig  list  to  the  Phase  II  ground  truth 
set  resulted  in  the  correct  identification  of  83%  of  the  TOI  recovered  (20  of  1 15  TOI  incorrectly 
classified  as  clutter).  Seventeen  of  the  missed  TOI  were  the  same  illumination  round  incorrectly 
identified  as  non-TOI  in  the  Phase  I  dig  list,  indicating  that  the  misclassification  of  this  item 
could  be  a  more  significant  problem  than  anticipated  following  Phase  I.  Retrospective  analysis 
of  the  dataset  indicated  that  these  items  could  be  identified  as  TOI  by  adding  examples  to  the 
classification  library,  but  doing  so  significantly  reduced  the  amount  of  clutter  that  could  be 
consistently  identified  as  non-TOI.  More  consideration  should  be  given  to  exactly  how  these 
specific  munitions  might  impact  the  groundwater  issues  at  MMR  and  how  many  added  digs 
would  be  necessary  to  detect  them  consistently. 
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1.0  INTRODUCTION 

Currently,  up  to  90  percent  of  excavation  costs  on  most  unexploded  ordnance  (UXO)  /  munitions 
and  explosives  of  concern  (MEC)  projects  are  related  to  removing  scrap  metal  that  does  not 
represent  an  explosive  hazard.  Significant  cost  savings  could  be  achieved  through  the  use  of 
geophysical  discrimination  methods  that  could  reduce  the  number  of  excavations  required  to 
remove  explosive  hazards  from  sites.  The  objective  of  this  project  is  to  demonstrate  the  use  of 
advanced  electromagnetic  induction  (EMI)  sensors  in  static  data  acquisition  mode  and  associated 
analysis  software.  To  achieve  these  objectives,  a  controlled  test  was  conducted  at  the 
Massachusetts  Military  Reservation  (MMR). 

This  is  one  of  a  series  of  the  Environmental  Security  Technology  Certification  Program  (ESTCP) 
demonstrations  of  classification  technologies  for  munitions  response  (MR).  This  demonstration 
is  designed  to  evaluate  classification  methods  at  a  site  that  is  known  to  contain  a  mix  of  munition 
types.  Munitions  of  interest  for  this  site  include  4.2-inch,  60-millimeter  (mm),  and  81 -mm 
mortars,  and  105 -mm  and  155-mm  projectiles. 

The  objectives  at  this  site  are  somewhat  different  from  the  standard  ESTCP  classification 
demonstration.  Most  MR  removal  actions  are  motivated  by  the  explosion  risk  of  the  UXO 
contamination.  In  this  case,  although  the  risk  of  explosion  is  present,  the  site  team  is  additionally 
concerned  about  the  potential  for  groundwater  contamination  from  the  energetic  filler  in  the 
UXO.  Because  of  this,  all  UXO  are  not  considered  equal  risks;  larger  projectiles  have 
substantially  more  filler  than  small  mortars  and  are  therefore  a  more  important  target. 

1.1  BACKGROUND 

The  Eiscal  Year  2006  defense  appropriation  contained  funding  for  the  “Development  of 
Advanced,  Sophisticated  Discrimination  Technologies  for  Unexploded  Ordnance  (UXO) 
Cleanup.”  The  ESTCP  responded  by  conducting  a  UXO  discrimination  study  at  the  former 
Camp  Sibert,  Alabama.  The  results  of  this  first  demonstration  were  very  encouraging.  The 
conditions  for  discrimination  were  favorable  at  this  site  and  included  a  single  target  of  interest 
(TOI;  4.2-inch  mortar)  and  benign  topography  and  geology.  All  of  the  classification  approaches 
demonstrated  were  correctly  identified  a  sizable  fraction  of  the  anomalies  as  arising  from 
nonhazardous  items  that  could  be  safely  left  in  the  ground.  Both  commercial  and  advanced 
sensors  produced  very  good  results.  ESTCP  chose  Camp  San  Luis  Obispo,  California,  as  the  site 
for  the  second  study,  which  provided  greater  challenges  in  topography  and  a  wider  mix  of  TOI. 
Again,  the  results  were  very  positive,  with  increased  discrimination  of  TOI  versus  nonhazardous 
items. 

In  2010,  the  third  ESTCP  study  was  conducted  at  the  former  Camp  Butner,  North  Carolina, 
which  included  smaller  TOI  than  either  of  the  previous  sites.  Great  success  was  achieved  in 
identifying  37-mm  projectiles,  fuzes,  and  larger  TOI  with  the  advanced  sensors.  ESTCP 
sponsored  a  fourth  study  in  201 1  at  the  former  Camp  Beale,  California,  a  site  with  a  wide  range 
of  TOI,  moderate  to  steep  terrain,  and  trees.  Previous  studies  included  open  field  pastures;  the 
study  area  at  Camp  Beale  included  medium-density,  wooded  areas  to  provide  increasing 
difficulty  to  test  the  high  standards  established  in  the  previous  studies.  Additionally,  the  Camp 
Beale  project  included  the  use  of  smaller,  man-portable  EMI  sensors  such  as  the  Naval  Research 
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Laboratory’s  Time-Domain  Electromagnetic  Multi-sensor  Towed  Array  Detection  System 
(TEMTADS)  2x2  cart,  Eawrence  Berkeley  National  Laboratory’s  man-portable  Berkeley  UXO 
discriminator  (BUD),  and  Sky  Research’s  man-portable  vector  machine.  All  of  the  EMI  sensors 
tested  at  the  former  Camp  Beale  were  quite  successful  in  discriminating  between  TOI  and 
clutter. 

The  study  area  at  MMR  was  chosen  by  ESTCP  to  test  MetalMapper’s  discrimination  capabilities 
in  areas  with  much  higher  target  densities  than  were  present  at  previous  demonstration  sites,  and 
its  effectiveness  in  identifying  large  target  items  within  high  target  densities. 

1.2  OBJECTIVES  OF  THE  DEMONSTRATION 

This  approach  has  the  potential  to  reduce  the  number  of  excavations  required  to  effectively 
remove  the  explosive  safety  risk  (MEC)  at  a  given  site,  which  would  result  in  significant  cost 
savings  related  to  the  closure  of  formerly  used  defense  sites.  The  cost  savings  are  expected  to  be 
particularly  significant  at  removal  action  sites.  Parsons  is  currently  involved  with  U.S.  Army 
Corps  of  Engineers  projects  at  several  MEC  and  recovered  chemical  warfare  materiel  sites  that 
could  be  used  for  additional  testing  and  refining  of  the  process  required  for  this  type  of 
discrimination  approach. 

1.3  REGULATORY  DRIVERS 

As  part  of  the  cleanup  of  former  Department  of  Defense  sites,  buy-in  is  required  from  regulatory 
agencies  at  the  federal,  state,  and  local  levels.  The  advancement  in  classification  sensors  and 
their  successful  deployment  at  real-world  sites  needs  to  be  documented  for  their  use  to  be 
accepted  by  the  applicable  regulatory  agencies.  Their  acceptance  of  the  use  of  this  technology  at 
sites  for  which  they  are  ultimately  responsible  will  be  particularly  important  because  of  the 
potential  for  Department  of  Defense  budget  cuts  to  affect  the  amount  of  money  that  will  be 
available  for  future  remedial  actions. 
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2.0  TECHNOLOGY 

2,1  TECHNOLOGY  DESCRIPTION 

The  MetalMapper  is  an  advanced  EMI  system  developed  by  Geometries,  Inc.,  with  support  from 
the  ESTCP.  The  MetalMapper  draws  elements  of  its  design  from  advanced  systems  currently 
being  developed  by  G&G  Sciences,  Inc.  (supported  by  Naval  Sea  Systems  Command,  the 
Strategic  Environmental  Research  and  Development  Program,  and  ESTCP)  and  by  Eawrence 
Berkeley  National  Eaboratory  with  support  from  the  Strategic  Environmental  Research  and 
Development  Program  and  ESTCP.  It  has  three  mutually  orthogonal  transmit  loops  in  the  Z,  Y, 
and  X  directions  and  contains  seven  triaxial  receiver  antennas  inside  the  Z  (bottom)  loop. 
Typically,  the  transmit  loops  are  driven  with  a  classical  bipolar  pulse-type  time  domain 
electromagnetic  waveform  (i.e.,  alternating  pulse  polarity  with  a  50%  duty-cycle).  Depending  on 
the  survey  mode  (e.g.,  Static/Dynamic),  the  fundamental  frequency  of  transmission  can  be  varied 
over  the  range  1.11  <f  <810  hertz.  The  seven  receiver  antennas  allow  21  independent 
measurements  of  the  transient  secondary  magnetic  field. 

The  data  acquisition  computer  (DAQ)  is  built  around  a  commercially  available  product  from 
National  Instruments.  The  National  Instruments  DAQ  is  a  full-featured  PC  running  Windows  7. 
The  DAQ,  electromagnetic  transmitter,  and  batteries  for  the  system  are  packaged  in  an  aluminum 
case  that  can  be  mounted  on  a  pack  frame,  on  a  separate  cart  such  as  a  hand  truck,  or  on  the 
survey  vehicle  such  a  tractor.  The  instrumentation  package  also  includes  two  external  modules 
that  provide  real-time  kinematic  (RTK)  global  positioning  system  location  and  platform  attitude 
(i.e.,  magnetic  heading,  pitch,  and  roll)  data.  These  modules  are  connected  to  the  DAQ  through 
serial  RS232C  ports.  A  block  diagram  of  the  DAQ  system  is  in  Eigure  2-1 . 

The  MetalMapper  has  two  modes  of  data  collection:  dynamic  and  static.  Data  collected  in 
dynamic  mode  results  in  data  files  containing  many  data  samples.  Generally  speaking,  dynamic 
mode  data  are  collected  while  the  antenna  platform  is  in  motion.  Static  mode  data  collection  is 
employed  for  cued  surveys.  As  its  name  implies,  the  antenna  platform  remains  static  or 
motionless  during  the  period  of  data  acquisition.  Depending  on  the  acquisition  parameters  (e.g., 
sample  period  and  stacking  parameter)  it  can  take  tens  of  seconds  to  complete  a  static 
measurement.  The  results  of  the  static  measurement  are  written  into  a  binary  data  file  containing 
only  a  single  data  point  representing  the  average  (stacked)  result,  usually  over  tens  or  even 
hundreds  of  repetitions  of  the  transmitter’s  base  frequency. 
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Figure  2-1:  Data  Acquisition  Computer  and  Functional  Block  Diagram 


A)  MM  DAQ  assembly  with  Panasonic  terminal. 


Data  are  acquired  in  time  blocks  that  consist  of  a  fixed  number  of  transmitter  cycle  “repeats.” 
Both  the  period  (T)  and  the  repeat  factor  (N)  are  operator  selectable  and  are  varied  in 
multiplicative  factors  of  3.  The  MetalMapper  also  averages  an  operator-specified  number  of 
acquisition  blocks  (NStacks)  together  before  the  acquired  data  are  saved  to  disk.  The  decay 
transients  that  are  received  during  the  off  times  are  stacked  (averaged)  with  appropriate  sign 
changes  for  positive  and  negative  half  cycles.  The  decays  in  an  individual  acquisition  block  are 
stacked,  and  the  decays  in  that  block  are  averaged  with  other  acquisition  blocks  (assuming  the 
operator  has  selected  NStack  greater  than  one).  The  resultant  data  are  saved  as  a  data  point.  A 
photo  of  the  typical  configuration  of  the  instrument  used  for  collecting  cued  data  is  shown  in 
Figure  2-2. 


Figure  2-2:  Antenna  Array  and  Deployment  of  the  MetalMapper  at  MMR 
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In  its  present  (third  generation)  form,  the  MetalMapper  teehnology  has  been  demonstrated  and 
seored  at  the  Standardized  UXO  Teehnology  Demonstration  sites  at  the  Yuma  Proving  Ground 
(blind  grid  only),  Aberdeen  Proving  Ground  (blind  grid  plus  direet  fire  and  indireet  fire  areas),  at 
Camp  San  Luis  Obispo  and  Camp  Butner  in  conneetion  with  2009  and  2010  classifieation 
studies,  and  at  Camp  Beale  during  2011  and  2012  live  site  demonstrations  earned  out  by  ESTCP. 
The  performance  of  the  MetalMapper  at  these  sites  is  documented  in  formal  reports  issued  by  the 
Aberdeen  Test  Center  and  by  the  various  demonstrators  who  analyzed  the  data  collected  at  Camp 
San  Luis  Obispo,  Camp  Butner,  and  Camp  Beale. 

Naval  Research  Laboratory  staff  also  used  the  laboratory’s  man-portable  TEMTADS  2x2x3 
array  to  collect  cued  data  on  approximately  1000  anomalies.  The  TEMTADS  system  was  used  to 
collect  data  over  300  anomalies  collected  by  the  MetalMapper  for  comparison.  The  TEMTADS 
data  results  are  presented  in  a  separate  report  authored  by  the  Naval  Research  Eaboratory  and  are 
not  covered  in  this  document. 

2,2  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

A  few  advanced  EMI  sensors  are  similar  to  the  MetalMapper  in  theory,  design,  and  size,  with  the 
most  comparable  being  the  TEMTADS  5x5  and  the  full-size  BUD.  The  TEMTADS  5x5  consists 
of  25  pairs  of  transmit/receive  coils  oriented  in  a  5x5  grid  pattern,  approximately  2  meters  to  a 
side.  The  BUD  consists  of  three  orthogonal  transmitters  and  eight  pairs  of  differenced  receivers. 
These  instruments  have  been  part  of  the  ongoing  ESTCP  classification  demonstrations,  and 
similar  results  have  been  documented  for  all  three  during  previous  projects.  The  main  advantage 
of  the  MetalMapper  is  that  it  is  currently  commercially  available,  while  the  other  two  advanced 
EMI  sensors  are  generally  only  used  by  the  organizations  that  developed  them.  As  discussed  in 
Section  1.1,  various  man-portable  EMI  sensors  were  tested  successfully  at  Camp  Beale.  As  with 
the  TEMTADS  5x5  and  the  BUD,  these  sensors  are  not  yet  commercially  available. 

The  greatest  limitation  of  the  MetalMapper  is  its  size,  both  of  the  sensor  itself  and  of  the 
accompanying  computer,  screen,  and  cables.  The  system  is  designed  primarily  for  use  in 
relatively  fiat,  open  fields  and  cannot  currently  be  used  effectively  in  wooded  areas. 
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3.0  PERFORMANCE  OBJECTIVES 

This  demonstration  had  three  primary  performance  objectives: 

•  Evaluating  whether  classification  techniques  will  work  at  the  MMR  site 

•  Evaluating  where  classification  techniques  will  work  at  MMR 

•  Evaluating  the  cost  effectiveness  of  classification  techniques  in  the  areas  at  MMR  where 
classification  is  determined  to  be  effective 

The  specific  performance  objectives  for  this  demonstration  are  summarized  in  Table  3-1. 

3.1  OBJECTIVE:  REPEATABILITY  OF  INSTRUMENT  VERIFICATION  STRIP 
MEASUREMENTS 

The  reliability  of  the  survey  data  depends  on  the  proper  functioning  of  the  survey  equipment. 
This  objective  concerns  the  twice-daily  confirmation  of  sensor  system  performance. 

3.1.1  Metric 

The  metrics  for  this  objective  were  the  distance  between  the  modeled  and  actual  locations  of  the 
seed  items  in  the  instrument  verification  strip  (IVS)  and  the  classification  results  for  each  point 
collected  over  an  IVS  seed. 

3.1.2  Data  Requirements 

Twice  daily  surveys  of  the  IVS  strip  were  used  to  judge  this  objective. 

3.1.3  Success  Criteria 

This  objective  was  met  if  the  modeled  X,  Y  locations  of  the  IVS  seed  items  are  within 
15  centimeters  (cm)  of  the  actual  locations,  if  the  depth  (Z  direction)  is  within  10  cm  of  the 
actual  burial  depth,  and  if  the  seeds  items  are  classified  as  the  correct  munition  type. 

3.2  OBJECTIVE:  SUCCESSFUL  REACQUISITION  OF  TARGETS  WITH  THE 
METALMAPPER 

Data  collected  directly  over  target  source  items  should  lead  to  a  high  probability  of  detecting  the 
TOI  at  the  site. 

3.2.1  Metric 

The  metric  for  this  objective  was  the  distance  between  the  center  of  the  MetalMapper  when  cued 
data  are  collected  and  the  modeled  location  of  the  source  item. 

3.2.2  Data  Requirements 

The  center  of  the  MetalMapper  was  determined  following  pre-processing  of  the  MetalMapper 
data  and  was  a  function  of  the  GPS  (Global  Positioning  System)  position  measured  when  the 
data  are  collected  and  the  attitude  (pitch  and  roll)  of  the  sensor.  The  modeled  location  of  the 
source  item  was  determined  following  initial  inversion  of  the  data. 
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3.2,3  Success  Criteria 

The  objective  was  considered  to  be  met  if  95%  of  the  collection  locations  (either  original 
collection  locations  or  re-shots  collected  due  to  excessive  offsets)  are  within  40  cm  of  the 
modeled  locations. 


Table  3-1:  Performance  Objectives  for  this  Demonstration 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Data  Collection  Objectives 

Repeatability  of 

IVS  measurements 

Measured  target  locations 
Correct  classification  of 

IVS  seeds 

•  Twice-daily  IVS  data 

X,  Y  <15  cm  (la) 

Z  <10  cm  (la) 

IVS  seed  items 
identified  correctly 

Successful 
reacquisition  of 
targets  with 
MetalMapper 

Offset  between  collection 
and  modeled  target 
locations 

•  GPS-located 
collection  location 

•  Modeled  target 
location 

95%  of  original  or  re¬ 
shot  target  locations 
<  40  cm  from 
modeled  locations 

Analysis  and  Classification  Objectives 

Maximize  correct 
classification  of 

TOI 

Number  of  TOI  retained. 

•  Prioritized  anomaly 
lists 

•  Scoring  reports  from 
IDA 

Approach  correctly 
classifies  all  TOI 

Maximize  correct 
classification  of 
non-TOI 

Number  of  false  alarms 
eliminated. 

•  Prioritized  anomaly 
lists 

•  Scoring  reports  from 
IDA 

Reduction  of  false 
alarms  by  >  50% 
while  retaining  all 

TOI 

Specification  of 
no-dig  threshold 

Probability  of  correct 
classification  and  number 
of  false  alarms  at 
demonstrator  operating 
point. 

•  Demonstrator  - 
specified  threshold 

•  Scoring  reports  from 
IDA 

Threshold  specified 
by  the  demonstrator  to 
achieve  criteria  above 

Minimize  number 
of  anomalies  that 
cannot  be  analyzed 

Number  of  anomalies  that 
must  be  classified  as 
“Unable  to  Analyze.” 

•  Demonstrator  target 
parameters 

Reliable  target 
parameters  can  be 
estimated  for  >  98% 
of  anomalies  on  each 
sensor’s  detection  list. 

Correct  estimation 
of  target 
parameters 

Accuracy  of  estimated 
target  parameters. 

•  Demonstrator  target 
parameters 

•  Results  of  intrusive 
investigation 

X,  Y  <15  cm  (la) 

Z  <10  cm  (la) 

o  =  standard  deviation;  cm  =  centimeter(s);  GPS  =  Global  Positioning  System;  IDA  =  Institute  for  Defense 
Analyses  IVS  =  instrument  verification  strip;  TOI  =  targets  of  interest 
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3.3  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  TARGETS  OF 
INTEREST 

One  of  the  two  main  objeetives  of  this  demonstration  was  to  correctly  classify  all  seeded  items 
and  any  MEC  items  remaining  at  the  site  as  TOI.  However,  while  the  goal  of  the  project  was  to 
identify  all  remaining  TOI  correctly,  the  ESTCP  Demonstration  Plan  (ESTCP,  2012)  did  note 
that  this  was  a  more  stringent  requirement  than  was  required  based  on  the  cleanup  objectives  for 
the  site.  Because  this  site  is  not  accessible  to  the  general  public  and  the  National  Guard  has  no 
future  plans  for  the  site  that  would  make  explosive  MEC  a  particular  hazard,  the  objectives  for 
this  site  are  more  focused  on  removing  MEC  based  on  its  potential  hazard  as  a  groundwater 
contamination  source  rather  than  for  the  explosive  hazard. 

3.3.1  Metric 

The  metric  for  this  objective  was  the  number  of  items  on  the  MetalMapper  anomaly  list  that  can 
be  correctly  classified  as  TOI. 

3.3.2  Data  Requirements 

Following  data  collection,  MetalMapper  data  was  analyzed  to  create  a  prioritized  dig  list,  which 
assigned  each  target  to  one  of  three  categories:  1)  TOI  2)  non-TOI,  or  3)  Can’t  Analyze.  The 
targets  classified  as  either  TOI  or  Can’t  Analyze  were  considered  “dig”  targets.  Institute  for 
Defense  Analyses  (IDA)  personnel  used  their  scoring  algorithms  to  assess  the  results. 

3.3.3  Success  Criteria 

The  objective  was  considered  to  be  met  if  all  of  the  items  of  interest  are  correctly  labeled  as  TOI 
on  the  prioritized  anomaly  list.  Given  the  requirement  to  remove  potential  sources  of 
groundwater  contamination,  results  for  the  site  were  deemed  adequate  if  a  large  percentage  of  the 
large  (60  mm  and  larger)  TOI  at  the  site  was  labeled  as  TOI  on  the  prioritized  anomaly  list. 

3.4  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  NON-TARGETS  OF 
INTEREST 

This  was  the  second  of  the  two  primary  measures  of  the  effectiveness  of  the  classification 
approach.  In  addition  to  correctly  classifying  TOI,  the  effectiveness  of  the  MetalMapper  in 
discriminating  munitions  was  a  function  of  the  degree  to  which  responses  that  do  not  correspond 
to  TOI  could  be  eliminated  from  consideration  during  the  intrusive  investigation. 

3.4.1  Metric 

The  metric  for  this  objective  was  the  number  of  targets  on  the  ranked  anomaly  list  created  using 
the  MetalMapper  data  that  could  be  correctly  classified  as  non-TOI. 

3.4.2  Data  Requirements 

Following  data  collection,  MetalMapper  data  was  analyzed  to  create  a  prioritized  dig  list,  which 
assigned  each  target  to  one  of  three  categories:  1)  TOI,  2)  non-TOI,  or  3)  Can’t  Analyze.  The 
targets  classified  as  non-TOI  were  considered  “no  dig”  or  non-TOI  targets.  IDA  personnel  used 
their  scoring  algorithms  to  assess  the  results. 
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3.4,3  Success  Criteria 

The  objective  was  considered  to  be  met  if  more  than  50%  of  the  non-TOI  items  can  be  correctly 
labeled  as  non-TOI  while  retaining  all  of  the  TOI  above  the  dig  threshold. 

3.5  OBJECTIVE:  SPECIFICATION  OF  NO-DIG  THRESHOLD 

In  a  retrospective  analysis,  as  performed  in  this  demonstration,  it  is  possible  to  tell  the  true 
classification  capabilities  of  a  classification  procedure  based  solely  on  the  ranked  anomaly  list 
submitted.  In  a  real-world  scenario,  not  all  targets  may  be  dug,  so  the  success  of  the  approach 
depended  on  the  ability  of  an  analyst  to  accurately  specify  their  dig/no-dig  threshold. 

3.5.1  Metric 

The  probability  of  correct  classification,  Pdass,  and  number  of  false  alarms,  Nfa,  at  the  dig/no  dig 
threshold  in  the  prioritized  dig  list  were  the  metrics  for  this  objective. 

3.5.2  Data  Requirements 

Following  data  collection,  MetalMapper  data  was  analyzed  to  create  a  prioritized  dig  list,  which 
assigned  each  target  to  one  of  three  categories:  1)  TOI,  2)  non-TOI,  or  3)  Can’t  Analyze.  The 
category  into  which  each  target  was  placed  determined  using  a  decision  statistic  to  be  developed 
during  analysis  of  the  MetalMapper  data.  The  dig/no  dig  threshold  for  this  project  was  the 
decision  statistic  value  that  separates  targets  classified  as  TOI  from  those  classified  as  non-TOI. 
IDA  personnel  used  their  scoring  algorithms  to  assess  the  results. 

3.5.3  Success  Criteria 

The  objective  was  considered  to  be  met  if  more  than  50%  of  the  non-TOI  items  were  correctly 
labeled  as  non-TOI  while  retaining  all  of  the  TOI  at  the  specified  threshold. 

3.6  OBJECTIVE:  MINIMIZE  NUMBER  OF  ANOMALIES  THAT  CANNOT  BE 
ANALYZED 

Anomalies  for  which  reliable  parameters  could  not  be  estimated  using  the  collected 
MetalMapper  data  cannot  be  classified.  These  anomalies  were  placed  in  the  dig  category,  which 
reduces  the  effectiveness  of  the  classification  process. 

3.6.1  Metric 

The  number  of  anomalies  for  which  reliable  parameters  cannot  be  estimated  was  the  metric  for 
this  objective. 

3.6.2  Data  Requirements 

Those  targets  for  which  parameters  could  not  be  reliably  estimated  were  identified  as  such  on  the 
prioritized  dig  list  submitted  following  analysis  of  the  MetalMapper  data. 

3.6.3  Success  Criteria 

The  objective  was  considered  met  if  reliable  parameters  can  be  estimated  for  >  98%  of  the 
targets  on  the  prioritized  dig  list. 
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3.7  OBJECTIVE:  CORRECT  ESTIMATION  OF  TARGET  PARAMETERS 

This  objective  involves  the  accuracy  of  the  target  parameters  that  are  estimated  in  the  first  phase 
of  the  analysis.  Successful  classification  is  only  possible  if  the  input  features  are  internally 
consistent.  The  obvious  way  to  satisfy  this  condition  is  to  estimate  the  various  target  parameters 
accurately. 

3.7.1  Metric 

Accuracy  of  estimation  of  target  parameters  is  the  metric  for  this  objective. 

3.7.2  Data  Requirements 

Target  parameters  were  provided  as  part  of  the  final  results  submission.  Parsons  compared  the 
submitted  modeled  source  locations  to  those  measured  during  the  intrusive  investigation. 

3.7.3  Success  Criteria 

The  objective  was  considered  to  be  met  if  the  estimated  X,  Y  locations  are  within  15  cm  (1 
standard  deviation  [a]),  and  the  estimated  depths  are  within  10  cm  (la). 
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4.0  SITE  DESCRIPTION 

MMR  was  chosen  as  the  next  in  a  progression  of  inereasingly  more  eomplex  sites  for 
demonstration  of  the  elassifieation  proeess.  The  first  site  in  the  series,  former  Camp  Sibert  in 
Alabama,  had  only  one  TOI,  and  item  “size”  was  an  effeetive  diseriminant.  The  eonditions  for 
diserimination  were  favorable  at  this  site  and  ineluded  a  single  TOI  (4.2-ineh  mortar)  and  benign 
topography  and  geology.  Additional  demonstrations  were  eondueted  at  Camp  San  Luis  Obispo, 
CA;  Camp  Butner,  NC;  the  Pole  Mountain  Target  and  Maneuver  Area,  WY;  Fort  Sill,  OK;  and 
Camp  Beale,  CA.  These  sites  all  added  eomplieations  sueh  as  an  inereased  number  of  munitions 
types,  ineluding  smaller  munitions  generally  more  diffieult  to  identify  from  munitions  fragments, 
and/or  more  diffieult  topographie  (i.e.,  hills  and  trees)  or  geologie  (rook  outoroppings) 
eonditions. 

MMR  is  an  approximately  200,000-aore  site  in  Western  Cape  Cod,  Massaohusetts.  The  seeding 
and  demonstration  were  eondueted  in  two  separate,  3-aore  areas  (northern  and  southern)  of  the 
Central  Impaot  Area  (CIA).  MetalMapper  data  were  only  oolleoted  in  the  southern  area; 
TEMTADS  data  were  primarily  oolleoted  in  the  northern  area,  although  TEMTADS  data  were 
also  oolleoted  over  300  targets  in  the  southern  area  to  provide  an  overlap  with  a  portion  of  the 
MetalMapper  targets.  An  aerial  photo  of  the  demonstration  area  is  shown  in  Eigure  4-1.  A  more 
speoifio  site  desoription  oan  be  found  in  the  Final  Central  Impact  Area  Source  Investigation 
Report  (Tetra  Teoh  EC,  Ino.  [Tetra  Teoh],  2011). 

4.1  SITE  SELECTION 

The  study  area  at  MMR  was  ohosen  by  ESTCP  to  test  the  oapabilities  of  the  MetalMapper  in 
areas  with  muoh  higher  target  densities  than  were  present  at  previous  demonstration  sites.  In 
addition,  the  terrain  in  this  study  area  was  generally  mueh  more  rugged  than  that  on  previous 
sites,  with  many  large,  deep  eraters  and  a  eonsiderable  amount  of  roughly  out  brush.  The  study 
area  boundaries  overlap  CIA  targets  T43,  T27,  T26,  and  Tl.  This  looation  was  ohosen  to 
inoorporate  this  work  as  part  of  the  National  Guard  Bureau’s  Impaot  Area  Groundwater  Study 
Program  (lAGWSP). 

4.2  BRIEF  SITE  HISTORY 

Portions  of  MMR  were  used  by  the  military  beginning  in  the  early  1900s.  The  CIA  has  been 
used  as  an  impaot  area  for  artillery  and  mortars  from  the  late  1930s  until  1997.  During  the  late 
1940s,  the  CIA  also  oontained  Navy  air-to-ground  rooket  ranges  that  utilized  2.25-inoh  rookets. 
Various  types  of  munitions,  inoluding  37-mm,  40-mm,  75-mm,  90-mm,  105-mm,  and  155-mm 
artillery  projeotiles  and  50-mm,  60-mm,  70-mm,  81 -mm,  3-inoh,  and  4.2-inoh  mortars,  have  been 
fired  into  the  CIA.  These  munitions  inolude  high  explosive  (HE)  oharges  designed  to  explode 
upon  impaot,  and  praotioe  or  “inert”  rounds  whioh  do  not  oontain  an  HE  oharge  but  may  oontain 
a  spotting  oharge  designed  to  emit  smoke  upon  impaot. 
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Figure  4-1:  Locations  of  Two  Demonstration  Study  Areas  within  Central  Impact  Area,  Massachusetts  Military  Reservation 
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The  predominant  HE  charge  used  in  pre -World  War  II  munitions  contained  TNT 
(trinitrotoluene).  Post-World  War  II  artillery  and  mortar  munitions  used  Composition  B  for  the 
HE  charge,  which  is  a  mixture  of  cyclotrimethyl enetrinitramine  and  TNT.  The  low-intensity 
training  round  artillery  practice  projectile  was  introduced  in  1982  to  reduce  the  noise  associated 
with  HE  explosions,  since  this  noise  was  a  source  of  complaints  from  the  public  abutters.  The 
low-intensity  training  round  includes  a  spotting  charge  containing  perchlorate.  The  use  of  HE 
artillery  projectiles  was  discontinued  in  1989,  and  the  firing  of  all  munitions  into  the  CIA  was 
discontinued  in  1997. 

HE  munitions  that  did  not  explode  (UXO)  or  that  partially  functioned  (UXO  low  order)  have 
accumulated  within  the  CIA  during  its  use.  UXO  along  roadways  or  at  other  locations  that 
presented  a  safety  hazard  due  to  human  access  have  historically  been  blown  in  place  using  an 
explosive  donor  charge.  Blow-in-place  operations  were  also  used  to  clear  areas  for  site 
investigation  under  the  lAGWSP  starting  in  1997.  Post-blow-in-place  soil  sampling  and  removal 
of  soil  contaminated  by  blow-in-place  activities  have  been  conducted  since  1999  under  the 
lAGWSP. 

4.3  MUNITIONS  CONTAMINATION 

The  suspected  munitions  in  the  CIA  include: 

•  5-inch,  7-inch,  8-inch,  14.5-mm,  20-mm,  30-mm,  37-mm,  75-mm,  90-mm,  105-mm,  155- 
mm,  and  1 75-mm,  projectiles 

•  60-mm,  81 -mm,  and  4.2-inch  mortars 

•  2.36-inch,  2.75-inch,  and  3.5-inch  rockets 

•  57-mm  recoilless  rifle  rounds 

The  munitions  of  primary  interest  for  the  demonstration  area  include  4.2-inch  mortars,  60-mm 
mortars,  81-mm  mortars,  105-mm  projectiles,  and  155-mm  projectiles.  These  larger  munitions 
contain  a  high  percentage  of  the  mass  of  explosives  remaining  in  the  CIA. 

4.4  SITE  CONFIGURATION 

The  demonstration  site  consists  of  two  separate,  3 -acre  locations  within  the  CIA  area, 
surrounding  targets  T43,  T27,  T26,  and  TI.  The  study  used  EM6I-MK2  data  previously 
collected  by  Tetra  Tech  to  identify  targets  for  survey  by  the  MetalMapper  and  TEMTADS  and  to 
identify  potential  locations  for  seeds  items.  The  demonstration  area  is  shown  in  Eigure  4-2. 
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Figure  4-2:  Demonstration  Site  Boundaries 
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5.0  TEST  DESIGN 

5.1  CONCEPTUAL  EXPERIMENTAL  DESIGN 

The  objective  of  this  program  is  to  demonstrate  a  method  for  the  use  of  classification  in  the  MR 
process.  The  three  key  components  of  this  method  are  collection  of  high-quality  geophysical 
data  and  principled  selection  of  anomalous  regions  in  those  data;  analysis  of  the  selected 
anomalies  using  physics-based  models  to  extract  target  parameters  such  as  size,  shape,  and 
materials  properties;  and  the  use  of  those  parameters  to  construct  a  ranked  anomaly  list.  Each  of 
these  components  will  be  handled  separately  in  this  program. 

The  National  Guard  Bureau  coordinated  the  EM61-MK2  cart  survey  data  collection  as  part  of 
their  ongoing  lAGWSP  and  provided  ESTCP  with  the  raw  data  for  this  demonstration.  ESTCP 
coordinated  preprocessing  of  the  survey  data  and  anomaly  selection,  to  create  a  master  anomaly 
list. 

Parsons  was  provided  the  anomaly  list  from  the  EM61-MK2  survey  .  Cued  MetalMapper  data 
were  collected  over  all  targets  on  the  master  anomaly  list  to  extract  target  parameters.  These 
parameters  were  passed  to  a  classification  routine  which,  after  training  on  a  limited  amount  of 
site-specific  ground  truth,  was  used  to  produce  a  prioritized  anomaly  list. 

Validation  digging  was  then  performed  for  all  anomalies  on  the  master  anomaly  list.  The 
underlying  target(s)  were  uncovered,  photographed,  located  with  a  cm-level  GPS  system,  and 
removed.  Ground  truth  data  was  then  requested  for  classification  training,  if  necessary. 

At  the  conclusion  of  training,  an  initial  ranked  anomaly  list  was  submitted.  This  list  was  ordered 
from  the  item  deemed  most  likely  to  be  a  munition  through  the  item  deemed  most  likely  to  be  not 
hazardous.  It  also  included  the  threshold  constituting  the  dig/no  dig  point  in  the  list.  Targets  for 
which  meaningful  parameters  could  not  be  extracted  were  placed  at  the  top  of  the  list.  Dig 
results  from  the  first  round  of  digging  were  provided  prior  to  the  construction  of  a  final  ranked 
anomaly  list,  with  each  anomaly  marked  dig  or  no-dig.  These  final  inputs  were  scored  by  the 
IDA  with  emphasis  on  the  number  of  items  that  are  correctly  labeled  nonhazardous  while 
correctly  labeling  all  TOT 

The  primary  objective  of  the  demonstration  was  to  assess  the  order  of  the  ranked  anomaly  list 
and  the  ability  to  specify  the  threshold  separating  high  confidence  clutter  from  all  other  items. 
The  secondary  objective  was  to  determine  the  classification  performance  that  could  be  achieved 
through  a  retrospective  analysis. 

5.2  SITE  PREPARATION 

5.2,1  Survey  of  Historical  Records 

Much  of  the  historical  information  on  this  site  has  been  summarized  in  the  Final  Central  Impact 
Area  Source  Investigation  Summary  Report  (Tetra  Tech,  2011).  These  reports  are  posted  on  the 
ESTCP  ftp  server  and  can  be  used  for  reference. 


15 


FINAL 


5.2.2  Acquire  Site-Specific  Information 

The  demonstration  site  falls  within  the  CIA.  The  range  has  formerly  been  deseribed  as  being 
used  for  40-mm  grenade,  66-mm  light  antitank  weapon,  and  2.36-inoh  and  3.5-inoh  rocket 
weapons  system  training.  These  weapons,  along  with  those  listed  in  Section  4.3,  were  recovered 
during  the  remedial  investigation  and  feasibility  study  or  are  suspected  to  be  present  at  the  site. 
Due  to  the  historical  usage  of  this  site  over  many  years,  it  is  also  likely  that  other  munitions  types 
beyond  those  listed  above  may  be  encountered. 

5.2.3  First-Order  Navigation  Points 

It  is  important  that  all  survey  data  and  validation  activities  be  conducted  on  a  common 
coordinate  system.  Therefore,  for  consistency.  Parsons  used  the  control  point  located  at  the 
range  control  office,  which  was  also  used  by  the  previous  digital  geophysical  mapping  team  and 
other  demonstrators.  Because  the  control  point  had  a  full-time  base  station  provided  by  Tetra 
Tech,  Parsons  synchronized  to  their  frequency  to  create  a  secondary  local  station  closer  to  the 
demonstration  areas.  The  coordinates  for  base  station  point  are  given  in  Table  5-1. 


Table  5-1:  Geodetic  Control  Location 


ID 

Latitude 

WGS  84 

Longitude 

WGS  84 

Elevation  (ft) 
from  Top  of 
Well  Casing 

LW4468 

41  42  30.92394N 

070  33  56.81178 

186.73 

5.2.4  Brush  Cutting  /  Surface  Clearance 

Significant  brush  cutting  and  a  surface  sweep  were  performed  by  Tetra  Tech  prior  to  their 
collection  of  the  EM61-MK2  data  used  to  select  targets  for  the  advanced  EMI  surveys.  The 
main  objective  of  the  surface  clearance  was  to  ensure  that  no  hazardous  items  would  be 
encountered  prior  to  the  nonintrusive  phases  in  the  demonstration  area  and  to  remove  metallic 
surface  debris  from  the  grids. 

5.2.5  Initial  EM61-MK2  100%  Coverage  Survey 

A  100%  coverage  EM61-MK2  survey  was  performed  by  Tetra  Tech  in  both  the  northern  and 
southern  survey  areas  prior  to  seeding  and  the  collection  of  advanced  EMI  data.  These  surveys 
were  used  to  guide  the  seeding  plan  and  to  identify  targets  for  the  advanced  EMI  surveys. 

5.2.6  Seeding  Operation 

At  a  live  site  such  as  this,  the  ratio  of  clutter  to  TOI  is  such  that  only  a  small  number  of  TOI  may 
be  found  in  the  investigation  area;  far  from  enough  to  determine  classification  performance  with 
acceptable  confidence  bounds.  To  avoid  this  problem,  the  site  was  seeded  with  enough  TOI  to 
ensure  reasonable  statistics. 

Parsons  conducted  seeding  operations  at  MMR  from  June  6,  2012,  through  June  10,  2012.  The 
seeding  operation  covered  approximately  6  acres  in  the  two  sections  of  the  CIA  selected  for  use 
in  the  demonstration.  The  location  of  each  seed  item  was  established  with  a  Trimble  R8  RTK 
GPS  system.  The  project  coordinate  system  was  in  Universal  Transverse  Mercator  (UTM)  Zone 
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19N,  WGS84  coordinates.  All  seed  items,  with  the  exeeption  of  medium  industry  standard 
objeets  (ISOs),  were  reeeived  from  the  Army  Researeh  lab  in  Weleome,  Maryland. 

Parsons  flagged  160  loeations  with  the  Trimble  R8  RTK  GPS  system  and  established  anomaly 
avoidanee  at  eaeh  loeation  to  ensure  a  clean  area  for  emplaeement.  All  160  seed  loeations  were 
dug  to  proper  size  and  depth.  Digging  operations  involved  both  meehanical  and  manual 
proeedures  to  meet  exaet  speeifieations  and  to  minimize  burial  evidenee.  Digging  operations 
proved  very  ehallenging  due  to  the  roots  and  fallen  trees  left  from  the  brush  eutting.  Heavy 
equipment  was  used  to  remove  large  tree  limbs  and  roots  prior  to  seeding.  Eaeh  seed  hole  was 
dug  to  meet  the  azimuth  and  depth  requirements  given  to  Parsons.  The  dip  angle  speeifieations 
were  set  to  a  45 -degree  toleranee  in  whieh  the  exaet  angles  was  determined  by  Parsons  but 
measured  with  a  level  for  doeumentation.  Exaet  angles  above  horizontal,  below  horizontal  were 
recorded.  After  all  the  emplaeement  requirements  of  depth,  inelination,  dip  angle,  length,  and 
loeation  were  eompleted,  a  photo  was  taken  of  the  seed  item  in  the  burial  location.  All  the 
emplaeement  information,  along  with  the  seed  item  and  north  direetion,  is  visible  in  the  photos. 

Seed  loeation  holes  were  not  backfilled  until  final  quality  eontrol  (QC)  eheeks  were  eomplete. 
QC  eheeks  eonsisted  of  eomparing  the  loeation  with  the  original  designated  loeation;  eapturing 
the  eenter  loeation  of  the  emplaeed  seed  item  with  GPS;  and  eheeking  the  depth,  inelination,  and 
dip  angle  of  eaeh  seed  item.  Once  these  cheeks  were  eomplete,  the  hole  was  backfilled  with  a 
shovel  to  prevent  any  exeess  movement  of  the  seed  items. 

Seed  items  for  the  MMR  demonstration  projeet  included  155-mm  projeetiles,  81-mm  projeetiles, 
105-mm  projeetiles,  4.2-ineh  mortars,  and  medium  ISOs.  A  list  of  the  seed  items  emplaeed  for 
the  projeet  is  ineluded  in  Table  5-2. 

Table  5-2:  Massachusetts  Military  Reservation 
Demonstration  Seed  Items 


Seed  Item 

Total 

Medium  ISO 

20 

81  mm 

40 

4.2-ineh  mortar 

20 

105-mm  projeetile 

40 

155-mm  projectile 

40 

Total 

160 

5.2,7  Establish  an  Instrument  Verification  Strip  and  Training  Pit 

A  elean  area  for  use  as  the  IVS  was  identified  in  an  unused  sand  road  immediately  adjaeent  to 
the  eonnex  box  used  to  store  the  projeet  equipment.  The  seetion  of  road  used  for  the  IVS  was 
approximately  6  feet  wide  and  100  feet  long.  A  list  of  the  seed  items  plaeed  in  the  IVS  is 
ineluded  in  Table  5-3. 
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Table  5-3:  Instrument  Verification  Strip  Seed  Items 


IVS  item 

Description 

Depth 

(cm) 

Orientation 

T-OOl 

Shot  put 

30 

N/A 

T-002 

155-mm  projectile 

50 

Horizontal 

T-003 

81 -mm  mortar 

30 

Horizontal 

T-004 

Blank  space 

N/A 

N/A 

T-005 

Medium  ISO 

30 

Horizontal 

5.3  SYSTEM  SPECIFICATION 

The  MetalMapper  sensor  and  data  aequisition  system  are  deseribed  in  detail  in  Section  2.1. 
During  the  demonstration  at  MMR,  the  antenna  array  was  placed  in  a  wooden  sled  attached  to 
the  rear  three-point  hitch  of  a  tractor  (Figure  2-2).  A  Trimble  R8  GPS  was  mounted  directly 
above  the  sensor  array  using  a  wooden  tripod,  and  an  inertial  measurement  unit  was  attached  to 
the  wooden  support  used  to  stabilize  the  X-  and  Y-direction  transmitters,  also  directly  above  the 
center  of  the  array.  These  instruments  streamed  positional  data  constantly,  at  a  rate  of 
approximately  10  hertz,  although  the  stream  rate  for  the  GPS  was  modified  throughout  the 
project  in  an  attempt  to  solve  repeated  software  crashes.  The  two  instruments  were  connected  to 
the  DAQ  via  USB  (universal  serial  bus)  ports,  and  the  incoming  data  were  used  both  to  navigate 
from  point  to  point  and  to  locate  the  collected  data. 

5.4  CALIBRATION  ACTIVITIES 

5.4.1  Test  Pit  and  Instrument  Verification  Strip  Data  Collection 

A  test  pit  was  constructed  at  the  site  before  the  arrival  of  the  MetalMapper  data  collection  team. 
The  pit  was  an  approximately  6-foot  by  3 -foot  by  4-foot  hole  that  allowed  the  collection  of  static 
MetalMapper  data  over  TOI  expected  at  the  site.  The  test  pit  data  could  then  be  used  for 
comparison  with  field  data  collected  over  unknown  targets.  Test  pit  items  were  generally 
oriented  in  four  directions  relative  to  the  MetalMapper  (vertical,  45  degrees  along  path, 
horizontal  along  path,  and  horizontal  perpendicular  to  the  path)  at  a  single  depth  selected  to 
produce  a  strong  signal-to-noise  ratio  for  each  orientation  of  the  test  object.  TOI  placed  in  the 
test  pit  were  intact,  inert  munitions  items  received  from  the  Army  Research  lab  or  recovered  at 
the  MMR  site  during  the  surface  sweep  or  previous  MMR  munitions  removal  projects. 
MetalMapper  data  were  collected  over  the  following  items: 

•  Medium  ISO 

•  8 1 -mm  mortar 

•  4.2-inch  mortar 

•  1 05 -mm  projectile 

•  105 -mm  high-explosive  antitank  (HEAT)  projectile 

•  155  -mm  proj  ectile 
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In  addition  to  the  test  pit  data,  data  were  eollected  over  the  IVS  twiee  daily.  The  items  in  the 
IVS  are  identified  in  Table  5-3.  All  data  eolleeted  over  the  IVS  strip  were  inverted  in  the  field  as 
deseribed  in  Seetion  6.2  and  compared  to  the  MMR  target  library  as  described  in  Section  6.4. 
Two  tests  were  performed  using  the  IVS  data: 

•  Inverted  locations  were  compared  to  the  known  locations  for  the  IVS  seed  items,  with  the 
differences  between  the  modeled  and  known  locations  expected  to  be  less  than  15  cm 
X,Y  and  10  cm  depth;  and 

•  The  item  identified  by  the  target  library  comparison  was  compared  to  the  actual  buried 
item,  and  it  was  expected  that  the  identified  item  matched  the  TOI  with  a  relatively  high 
confidence  (0.7  weighted  metric  confidence  expected  in  the  field).  Identified  results  were 
considered  a  match  to  the  IVS  seed  as  long  as  the  sizes  of  the  two  items  were  relatively 
similar  (e.g.,  medium  ISO  seed  identified  as  a  60-mm  projectile  was  acceptable,  shot  put 
identified  as  an  81 -mm  mortar  was  not). 

IVS  testing  results  are  detailed  in  Section  7.1. 

5.4,2  Background  Data 

Given  the  high  density  of  targets  within  the  survey  area,  background  data  could  not  be  collected 
efficiently  within  the  grids  because  background  locations  could  not  be  identified  either  in  the 
EM61-MK2  data  or  by  maneuvering  the  MetalMapper  in  the  survey  area.  Therefore, 
background  data  from  the  IVS  strip  (location  T-004)  were  used  to  correct  each  day’s  survey  data. 
The  background  point  used  to  correct  data  (i.e.,  morning  or  afternoon)  was  determined  using  the 
point  collected  closest  in  time  to  the  point  being  corrected.  Generally,  each  IVS  point  was  used 
to  correct  approximately  half  of  the  points  collected  each  day. 

5.5  DATA  COLLECTION  PROCEDURES 

The  operator  moved  the  array  by  lifting  the  sled,  navigating  to  the  vicinity  of  each  selected  point 
using  the  graphic  display  on  the  computer  monitor  in  front  of  him,  and  setting  the  MetalMapper 
down  on  the  point.  Reacquisition  of  the  EM61-MK2  targets  selected  for  cued  data  collection 
was  accomplished  using  “dancing  arrows”  displayed  on  the  monitor.  The  “dancing  arrows” 
display  shows  the  seven  receivers  in  the  array,  arranged  as  they  are  in  the  Z-coil,  typically  with  a 
blue  arrow  pointing  out  of  each.  The  arrows  point  toward  the  metallic  source  nearest  each  of  the 
receivers.  Under  ideal  conditions,  there  is  one  source  in  the  vicinity  of  the  selected  point,  and  all 
of  the  arrows  point  inward  toward  the  center  of  the  array.  In  the  case  of  multiple  sources,  one  or 
more  of  the  outer  arrows  may  point  outward  from  the  array  toward  another  piece  of  metal. 
Generally,  the  operator  attempted  to  position  the  array  such  that,  at  least,  the  arrows  in  the  three 
receivers  closest  the  middle  of  the  coil  were  pointing  at  each  other. 

The  MetalMapper’s  single-point  or  cued-collection  mode  was  used  for  all  data  collection  at 
MMR.  Once  the  MetalMapper  was  positioned  correctly  above  the  target,  the  operator  collected  a 
data  point  using  the  settings  indicated  in  Table  5-4. 
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Table  5-4:  Acquisition  Parameters  Used  during  the  Camp  Beale  Demonstration 


Mode 

Tx 

Mode 

Hold-Off 

Time 

(jis) 

Block 

Period(s) 

Rep 

Fctr 

Dec 

Fctr 

(%) 

Stk 

Const 

Base 

Freq 

(Hz) 

Decay 
Time  (ps) 

No. 

Gates 

Sample 
Period  (s) 

Sample 
Rate  (S/s) 

Static 

ZYX 

50 

0.9 

27 

10 

10 

30 

8,333 

50 

9 

N/A 

|j,s  =  microsecond(s);  Hz  =  hertz;  s  =  second(s);  S/s  =  samples  per  second 


Static  targets  were  identified  according  to  the  identification  determined  for  each  target  picked  in 
the  dynamic  EM61-MK2  survey.  In  the  case  of  repeated  measurements  associated  with  a  single 
target  point,  10,000  was  added  to  the  original  identification  (e.g.,  the  re-shot  for  0001  was 
10001). 


5.5.1  Scale  of  Demonstration 

Parsons’  field  team  eollected  2,312  data  points  during  the  project  for  2,271  targets.  Sixteen  of 
the  2,287  targets  selected  in  the  EM61-MK2  data  could  not  be  collected  due  to  vegetation  or 
terrain.  Additional  points  were  re-shots  of  already  collected  points  due  to  a  high  offset  between 
the  location  of  the  center  of  the  MetalMapper  and  the  modeled  location  of  a  target,  points  due  to 
a  high  offset  between  the  loeation  identified  in  the  EM61-MK2  survey  and  the  center  of  the 
MetalMapper,  and  points  that  may  have  been  ineorrectly  identified  during  initial  eolleetion.  A 
distance  of  40  cm  between  the  array  location  and  the  modeled  location  was  considered  the 
greatest  acceptable  distance  between  those  two  points.  The  greatest  acceptable  distance  between 
the  EM61-MK2  pick  and  MetalMapper  eolleetion  point  was  73  cm,  which  was  based  on  the 
standard  USAGE  measurement  quality  objective  of  14  of  the  survey  line  spacing  (75  cm  for  the 
EM61-MK2  survey)  plus  35  cm  for  allowable  offset  between  seed  items  and  target  selection 
locations.  Re-shots  were  collected  for  any  targets  with  larger  offsets. 

5.5.2  Sample  Density 

One  data  point  was  collected  per  target,  as  described  in  Section  5.5;  re-shots  were  colleeted  for 
targets  with  modeled  loeations  greater  than  40  cm  from  the  collection  location  or  for  collection 
locations  farther  than  73  cm  from  the  EM61-MK2  pick. 

5.5.3  Data  Quality  Checks 

An  instrument  calibration  check  was  conducted  at  least  twice  a  day  (at  the  beginning  and  the  end 
of  the  field  day).  These  checks  ensured  that  the  instrumentation  was  functional,  properly 
calibrated,  and  stable. 

A  final  cheek  on  the  quality  of  static  data  was  performed  after  initial  inversion  was  performed 
using  the  UX- Analyze  module  in  Oasis  montaj.  Inverted  target  locations  were  compared  to  data 
collection  locations  to  determine  if  offsets  between  the  two  are  greater  than  40  em.  The 
eolleetion  location  was  also  compared  to  the  location  of  the  EM61-MK2  target.  Re-shots  were 
collected  for  targets  with  collection  to  modeled  locations  greater  than  40  cm  or  EM61-MK2  pick 
to  collection  locations  distances  greater  than  73  cm. 
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5.5,4  Data  Handling 

Data  were  reeorded  in  binary  format  as  files  on  the  hard  disk  of  the  MetalMapper  DAQ.  These 
data  were  offloaded  to  other  media  at  least  onee,  and  sometimes  more  frequently,  per  day.  The 
eomputer’s  hard  disk  had  enough  eapaeity  to  store  all  the  data  from  the  entire  site,  so  these  data 
were  not  erased  until  they  had  been  thoroughly  reviewed  and  archived.  The  data  file  names 
acquired  each  day  were  cataloged  (usually  on  a  spreadsheet)  and  integrated  with  any  notes  or 
comments  in  the  operator’s  field  book.  All  data  ended  up  on  the  hard  drives  of  one  or  more 
laptop  computers  used  to  post-process  data.  Data  were  also  archived  to  a  data  server  in  the 
Parsons  office. 

Raw  binary  fdes  were  preprocessed  using  the  TEM2CSV  software  package,  which  outputs 
“preprocessed  and  located”  data  fdes  in  a  text  readable  format  (.csv).  Preprocessing  included  the 
location  of  the  point  in  UTM  meters  and  subtraction  of  background.  Located  and  background- 
corrected  .CSV  files  were  imported  into  Oasis  montaj  for  further  processing  and  analysis. 

5.6  INTRUSIVE  PROCEDURES 

Intrusive  operations  were  broken  down  into  two  phases,  I  and  II.  Phase  I  was  performed 
between  July  17,  2012,  and  August  10,  2012,  and  included  the  investigation  of  all  the 
TEMTADS-only  grids  in  the  northern  area  and  a  signification  portion  of  the 
TEMTADS/MetalMapper  and  MetalMapper-only  grids  in  the  southern  area.  A  total  of  1,362 
targets  were  investigated  during  Phase  I,  including  all  of  the  TEMTADS-only  targets  that  could 
be  investigated,  all  300  of  the  combined  TEMTADS/MetalMapper  targets,  and  approximately 
600  MetalMapper-only  targets.  The  only  TEMTADS-only  targets  not  investigated  were  within 
the  road  passing  through  these  grids,  and  no  intrusive  operations  were  performed  on  or  near  the 
road.  Phase  II  of  the  intrusive  investigation  took  place  between  May  20,  2013,  and  June  20, 
2013,  and  included  1,049  of  the  remaining  1,386  MetalMapper-only  targets  not  investigated 
during  Phase  I.  The  final  337  targets  were  not  excavated  due  to  time  constraints  at  the  site. 
Table  5-5  summarizes  the  targets  investigated  and  sources  recovered  by  intrusive  phase. 

Intrusive  operations  began  with  site-specific  training,  which  included  prepping  the  staging  area 
for  intrusive  activities  and  performing  equipment  checks.  The  staging  area  consisted  of  a 
20-foot-long  connex  box  approximately  %  mile  from  the  survey  area.  All  Parsons  intrusive 
equipment  was  stored  in  the  connex  and  locked  at  the  end  of  the  day.  Daily  equipment  checks 
included  confirming  GPS  accuracy  over  known  monuments,  EM61-MK2  static  tests,  and 
handheld  analog  instruments  calibrations. 

All  excavated  anomalies,  excluding  the  seed  items,  were  placed  in  a  sandbag  and  stored  in  the 
connex.  MEC  items  discovered  during  the  intrusive  operations  were  flagged  and  Tetra  Tech 
was  notified  immediately.  Parsons’  intrusive  team  worked  with  Tetra  Tech  UXO  personnel 
throughout  the  project  on  protocols  involving  MEC  and  overall  safety.  Parsons  also  kept  daily 
communications  with  range  control  and  the  U.S.  Army  Corp  of  Engineers  on  the  schedule  of  the 
impact  areas  and  planned  accordingly  to  work  around  live  firing  operations.  Phase  II  intrusive 
operations  were  conducted  in  a  similar  manner  to  the  Phase  I  operations,  except  that  the 
recovered  items  from  each  target  location  were  not  bagged  and  stored  separately. 
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Table  5-5:  Intrusive  Phase  Summary 


Intrusive 

Phase 

Southern 

Grids 

Targets 

Investigated 

Southern 

Grids 

Sources 

Recovered 

Northern 

Grids 

Targets 

Investigated 

Northern  Grids 
Sources 
Recovered 

Phase  I 

901 

2,789 

431 

1,227 

Phase  II 

1,049 

5,109 

0 

0 

Personnel  on  site  to  eonduct  the  intrusive  operation  included  only  Parsons  with  Tetra  Tech 
supporting  demo  operations  on  a  separate  contract.  The  field  team  consisted  of  seven  Parsons 
personnel.  Parsons’  site  safety  and  health  manager  and  site  manager  conducted  daily  site  safety 
briefings,  as  appropriate. 

5.6.1  Equipment 

The  equipment  used  during  the  MMR  intrusive  activities  included  the  following: 

•  Schonstedt  Magnetic  Locator  (Model  GA-52Cx) 

•  Whites  Metal  detector 

•  EM61-MK2 

•  Trimble  R8  GPS  system 

•  Digital  level 

•  Miscellaneous  hand  tools 

•  Digital  cameras 

5.6.2  Field  Procedures 

Reacquisition  of  all  targets  was  conducted  using  the  Trimble  R8  GPS  system.  The  GPS  base 
station  was  set  up  on  survey  monument  established  using  the  Tetra  Tech  base  station.  Parsons 
flagged  all  target  locations  with  a  plastic  pin  flag  marked  with  the  target  identification  and 
EM61-MK2  pre-value.  The  depth  estimations  determined  the  initial  approach  to  every  target. 
To  preserve  the  azimuth  and  inclination  of  the  anomalies,  the  digging  of  all  targets  began  with 
skinning  the  surface  manually  by  hand.  Azimuth  data  reflected  magnetic  north;  inclination  was 
determined  using  a  digital  level  and  readings  reflected  positive  or  negative  from  horizontal. 
Eocation  data  captured  by  GPS  was  used  to  document  the  center  mass  and  elevation  of  each 
item.  A  photograph  was  collected  of  the  item  with  written  dig  result  data  on  a  whiteboard.  An 
EM61-MK2  unit  was  used  to  scan  the  location  to  confirm  the  absence  of  all  metallic  items  from 
that  target  location. 

The  Parsons  team  leader  who  orchestrated  the  movements  of  the  different  tasks  associated  with 
the  information-gathering  process  recorded  all  documentation  on  a  dig  sheet.  The  intrusive 
operations  consisted  of  two  intrusive  teams:  a  heavy-equipment  team  and  a  hand-dig  team,  based 
on  the  depth  and  difficulty  of  the  excavation.  As  the  two  teams  excavated  the  anomalies,  other 
Parsons  personnel  helped  with  the  documentation  and  GPS  tasks. 
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Munitions  debris  (MD)  and  cultural  debris  (CD)  scrap  collected  from  target  locations  was  stored 
in  labeled  burlap  sandbags  with  the  pin  flag.  Large  155-mm  projectiles  were  stored  in  a 
designated  area  determined  by  Tetra  Tech.  Parsons’  senior  UXO  and  site  safety  officer  worked 
with  Tetra  Tech  to  transfer  the  certified  MD  scrap  to  a  holding  location  used  by  Tetra  Tech  as 
part  of  their  ongoing  project.  All  seed  items  recovered  from  intrusive  operations  were  stored  in  a 
secure  area  and  prepared  for  final  shipment. 

All  target  locations  were  backfilled  after  completion  of  the  excavation.  After  the  final  anomalies 
were  excavated  and  backfilled,  Parsons  conducted  a  walkthrough  and  confirmed  that  all  holes 
were  filled  and  no  trash  was  left. 

Excavation  data  collected  by  the  intrusive  team  was  digitally  downloaded  to  a  database  and 
reviewed  daily.  The  daily  information  required  the  target  ID  to  be  connected  with  intrusive 
documentation,  photo,  and  GPS  coordinates.  Assessment  of  each  target  item  required  the 
coordinates  to  match  the  original  location  and  the  picture  to  match  the  documented  findings. 
Photographs  of  the  intrusive  operation  are  shown  in  Figure  5-1. 

5.7  INTRUSIVE  RESULTS 

5.7.1  Phase  I  Results 

During  the  Phase  I  intrusive  operation.  Parsons  investigated  1,376  anomalies.  These  included  all 
of  the  anomalies  in  the  northern  area  except  for  those  in  or  near  the  road,  and  895  in  the  southern 
area.  On  average,  86  digs  were  completed  per  day,  based  on  16  intrusive  days.  Given  multiple 
items  recovered  from  the  majority  of  the  anomaly  locations,  more  than  4,000  sources  were 
removed  during  the  investigation.  Most  of  the  MD  encountered  consisted  of  large  pieces  of 
munitions  fragments  and  empty  projectiles.  The  results  of  the  Phase  I  intrusive  investigation  are 
summarized  in  Table  5-6. 

5.7.2  Phase  II  Results 

Parsons  investigated  1,049  anomalies  in  the  southern  area  during  the  Phase  II  investigation.  As 
with  the  Phase  I  effort,  multiple  items  were  recovered  from  most  of  the  intrusive  locations,  and 
more  than  5,000  sources  were  removed  during  the  investigation.  On  average,  53  digs  were 
completed  per  day,  based  on  20  intrusive  days.  There  were  some  location  issues  during  the 
Phase  I  intrusive  investigation  due  to  the  proximity  of  sources  to  each  other  and  the  large  number 
of  items  pulled  from  each  hole,  so  more  care  was  taken  to  reduce  the  number  of  holes  open  at 
one  time  during  Phase  II.  This  was  likely  the  cause  of  the  discrepancy  between  the  dig  rates  for 
the  two  phases.  The  results  of  the  Phase  II  intrusive  investigation  are  summarized  in  Table  5-6. 
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Figure  5-1:  Intrusive  Operation  Photos 
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Table  5-6:  Intrusive  Results 


Type 

Targets 

Northern  Area 

Phase  I 

Targets  of  interest 

(munitions  and  explosives  of  ooncern,  material 
potentially  presenting  an  explosive  hazard,  seed,  intaot 
round  identified  as  munitions  debris) 

26 

Munitions  debris 

1,129 

Other  debris 

72 

No  eontaet 

0 

Southern  Area 

Phase  I 

Phase  II 

Targets  of  interest 

(munitions  and  explosives  of  eoneern,  material 
potentially  presenting  an  explosive  hazard,  seed,  intact 
round  identified  as  munitions  debris) 

126 

115 

Munitions  debris 

2,647 

4,977 

Other  debris 

13 

16 

No  eontaet 

3 

1 

5.8  MUNITIONS  DEBRIS  SCRAP 

MD  and  CD  scrap  recovered  from  the  demonstration  area  at  MMR  were  provided  to  the  onsite 
environmental  eontraetor.  The  MD/CD  serap  filled  six  55-gallon  drums,  all  with  looks  and 
numbered  custody  seals.  Disposition  of  the  reoovered  MD  was  performed  by  the  onsite 
environmental  eontraetor.  Recovered  seed  items  not  taken  from  onsite  stookpiles  of  MD  at  the 
beginning  of  the  project  were  returned  to  Glen  Harbaugh  at  the  Army  Researoh  Lab. 
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6.0  DATA  ANALYSIS  AND  PRODUCTS 

The  MetalMapper  was  used  to  colleet  statie  data  over  2,271  of  the  2,287  targets  identified  at 
MMR  based  on  EM61-MK2  data.  Sixteen  targets  were  not  eollected  because  they  were  either  in 
the  woods  or  on  top  of  a  berm  and  were  inaccessible  to  the  MetalMapper.  The  processing  and 
analysis  steps  that  were  used  to  generate  a  dig/no  dig  decision  for  each  target  are  described 
below. 

6.1  PREPROCESSING 

Raw  MetalMapper  data  are  collected  and  stored  as  .tern  files.  The  MetalMapper  acquisition 
software  uses  a  convention  for  assigning  a  unique  name  to  each  data  file  without  the  need  to 
manually  enter  the  name.  The  operator  supplies  a  prefix  for  the  root  name  of  the  file  (e.g., 
“Static”).  The  acquisition  software  then  automatically  appends  a  five-character  numerical  index 
to  the  filename  prefix  to  form  a  unique  root  name  for  the  data  file  (e.g.,  StaticOOOOl).  The  index 
is  automatically  incremented  after  the  file  has  been  successfully  written.  Although  the  target 
identification  is  not  used  as  the  file  name  in  the  .tern  file,  the  Target  identification  is  stored  in  the 
file  according  to  name  of  the  target  highlighted  on  the  MetalMapper  screen  during  collection. 

Preprocessing  of  the  .tern  files  was  accomplished  using  TEM2CSV,  a  program  specifically 
developed  for  this  purpose.  TEM2CSV  subtracted  the  site  background  from  the  data  point  using 
a  background  file  specified  by  the  user,  converted  the  points  from  the  geographic  coordinate 
system  used  for  collection  to  the  UTM  Zone  19N  coordinate  system  used  for  processing,  and 
exported  the  resulting  data  to  a  .csv  file  that  could  be  imported  into  The  UX- Analyze  package  in 
Geosoff  s  Oasis  montaj  software.  The  exported  .csv  file  name  contained  both  the  collection 
identification  and  the  Target  identification  (e.g.,  1651_Static00001_1651).  Preprocessing  was 
typically  completed  in  batches  based  on  time  to  account  for  differing  background  data.  At 
MMR,  the  survey  area  contained  so  much  subsurface  metal  that  no  background  locations  readily 
accessible  for  frequent  re-survey  could  be  identified  in  the  EM61-MK2  data.  Therefore,  data 
collected  at  the  background  IVS  point  (IVS-04)  in  the  morning  and  afternoon  were  used  to 
correct  all  of  the  survey  data.  Unless  there  appeared  to  be  a  problem  with  a  particular 
background  file,  data  were  tied  to  the  nearest  background  point  in  time. 

6.2  PARAMETER  ESTIMATION 

All  MetalMapper  data  points  were  inverted  using  UX-Analyze  to  determine  a  number  of 
modeled  parameters  for  each  target.  These  parameters  included  the  location,  size,  and 
orientation  of  the  source  object;  the  polarizability  of  each  axis  of  the  object;  and  information 
regarding  the  quality  of  the  data  and  the  relative  match  between  the  inverted  data  and  the 
expected  model. 

All  target  inversion  was  initially  performed  using  the  UX-Analyze  batch  processing  mode  with 
the  multiple  object  solver  enabled.  Targets  for  which  multiple  objects  were  identified  using  the 
multiple  object  solver  were  re-inverted  using  the  batch  processor  without  the  multiple  object 
solver  enabled.  In  these  cases,  the  single  object  and  multiple  object  results  were  compared  to 
determine  which  method  returned  a  result  more  indicative  of  TOT  Although  the  multiple  object 
result  may  have  approximated  the  expected  model  to  a  higher  degree,  the  result  more  indicative 
of  potential  TOl  was  used  for  target  ranking  to  be  conservative.  As  discussed  further  in  Section 
6.4.1,  initial  dig  list  results  suggested  that  the  UX-Analyze  multiple  object  solver  in  version  7.5 
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of  Oasis  Montaj  did  not  separate  out  multiple  objects  effectively  enough  for  some  of  the  targets 
at  MMR.  As  a  result,  all  of  the  data  were  re-inverted  using  the  updated  solver  in  Oasis  Montaj 
version  7.5  Tl.  The  main  difference  between  the  two  solvers  was  the  number  of  results  obtained 
for  each  target.  The  old  solver  returned  only  one  result  using  one  algorithm  to  determine 
whether  various  point  clouds  were  modeled  as  single  or  multiple  objects.  The  version  in  the  Tl 
build  returned  up  to  four  results  for  each  target  using  four  different  algorithms  to  organize  point 
clouds  into  various  combinations  of  single  or  multiple  objects. 

6.3  CLASSIFIER  AND  TRAINING 

6,3.1  Confidence  Metrics 

The  polarization  curves  developed  for  each  target  were  compared  to  a  library  of  known 
polarization  curves  compiled  using  test  stand  data  and  test  pit  data  from  MMR.  The  items  in  the 
MMR  comparison  library  were  limited  to  the  larger  TOI  expected  at  the  MMR  site:  60-mm, 
81-mm,  and  4.2-inch  mortars  and  105-  and  155-mm  projectiles.  Examples  of  various  types  of 
these  items  were  used  (e.g.,  105-mm  projectile  and  105-mm  HEAT  round),  but  items  not 
expected  at  the  site,  such  as  bombs,  were  not  included.  Additionally,  given  the  primary  goal  of 
the  demonstration  (i.e.,  the  identification  of  large  MEC  items  contain  relatively  large  amounts  of 
explosive),  smaller  MEC  such  as  40-mm  grenades  were  not  included  in  the  library  even  though 
they  were  known  to  be  present  on  site.  An  initial  comparison  between  the  measured  targets  and 
the  library  data  was  performed  using  a  weighted  confidence  metric  for  the  three  primary 
polarizabilities  (size:  1,  shape  1:  0.5,  shape  2:  0.5).  During  this  comparison,  the  four  results  from 
the  multiple  object  solver  were  compared  to  the  library,  and  the  one  with  the  highest  confidence 
metric  was  selected  for  use  with  that  target.  If  the  result  selected  was  not  the  one  already  in  the 
target  database,  the  database  result  was  replaced.  All  further  confidence  metrics  were  generated 
using  the  multiple  object  result  selected  during  the  weighted  metric  comparison.  In  addition  to 
the  weighted  confidence  metric  generated  during  the  initial  comparison  of  the  results  to  the 
library,  three  more  metrics  were  generated  for  each  target: 

1)  three-curve  metric  -  size:  1,  shape  1:1,  shape  2:  1 

2)  two-curve  metric  -  size:  1,  shape  1:1,  shape  2:  0 

3)  one-curve  metric  -  size:  1,  shape  1:0,  shape  2:0 

As  a  first  step  each  target  was  examined  by  looking  at  a  figure  showing  the  two  closest  matches 
for  the  weighted,  two-curve,  and  one-curve  comparisons  to  the  library.  Results  were  generally 
grouped  into  one  of  five  categories: 

1)  All  three  polarizability  curves  (pi,  P2,  P3)  were  usable 

2)  Only  two  of  the  curves  (pi,  P2)  were  usable 

3)  Only  pi  was  usable 

4)  No  usable  curves,  but  it  was  determined  unlikely  that  there  was  a  target  large  enough 
to  be  TOI  in  the  acquisition  location 

5)  Can’t  analyze  (no  usable  curves,  and  it  was  considered  likely  that  the  curves  were 
unusable  despite  the  existence  of  a  source  potentially  large  enough  to  be  TOI) 

The  difference  between  targets  deemed  likely  to  have  a  source  potentially  related  to  TOI  (can’t 
analyze)  and  those  with  unusable  curves  due  to  a  small  or  nonexistent  source  was  typically 
determined  based  on  the  modeled  location  of  the  source  relative  to  the  collection  point  and  the 
signal  strength  calculated  for  the  target  in  question.  If  the  source  was  modeled  within  40  cm  of 
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the  collection  location  and  the  signal  strength  was  less  than  20,  the  target  was  considered  likely 
to  be  a  no  contact.  The  assumption  was  that  the  targets  with  signal  amplitudes  higher  than  20 
were  considered  potentially  large  objects  for  which  the  data  might  be  poor.  In  addition,  the  field 
team  recorded  notes  in  their  logbook  for  all  targets  that  could  not  be  found  near  the  picked 
location  using  the  MetalMapper’s  dancing  arrows.  Targets  with  such  notes  were  removed  from 
consideration  as  can’t  analyze  targets. 

6.3.2  Training  Data 

Training  data  selection  for  MMR  began  with  a  comparison  of  the  library  data  to  the  survey  data 
using  two  decay  (time  gates  8-32  and  8-43)  versus  size  feature  space  plots  (Figures  6-1  and  6-2). 
A  preliminary  threshold  separating  targets  possibly  representing  TOI  and  targets  more  likely  to 
be  clutter  was  indentified  on  each  plot  based  on  the  location  of  the  library  data  on  the  plots.  The 
threshold  was  set  at  0.035  for  the  8-32  decay  plot  and  at  0.0045  for  the  8-43  decay  plot.  The 
initial  training  data  request  for  MMR  was  composed  of  25  targets  selected  for  their  similarity  to 
ordnance  based  on  the  library  match  and  on  their  location  within  the  space  plots.  All  of  the 
targets  selected  had  weighted  confidence  metrics  greater  than  0.575,  two-curve  metrics  greater 
than  0.700,  and  one-curve  metrics  greater  than  0.800,  suggesting  a  reasonably  good  match  to 
library  TOI.  Unlike  other  sites  in  the  live  site  demonstration  program,  a  large  percentage  of  the 
targets  at  the  site  fit  these  criteria.  Therefore,  many  of  the  training  data  selections  were  made  to 
see  if  targets  with  relatively  strong  matches  to  the  library  data  could  be  filtered  out  based  on 
decay.  Many  of  the  selected  targets  were  those  for  which  the  decay  for  gates  8-32  was  above  the 
0.035  threshold  but  the  decay  for  gates  8-43  was  below  the  0.004  threshold,  or  vice  versa. 
Additional  consideration  was  given  to  whether  there  appeared  to  be  any  clusters  of  targets  below 
either  of  the  thresholds. 

Only  two  of  the  requested  targets  were  TOI  (Figures  6-1  and  6-2),  both  relatively  large-sized 
targets  within  the  dataset.  In  addition,  both  were  below  the  gates  8-32  decay  threshold  and  one 
was  below  the  gates  8-43  threshold.  These  results  suggested  that  the  use  of  these  thresholds  as 
definitive  cutoffs  between  likely  TOI  and  likely  non-TOI  was  more  applicable  for  smaller 
sources  than  it  was  for  larger  sources. 

6.3.3  Decision  Statistic 

Classification  for  the  MMR  project  was  accomplished  using  three  of  the  confidence  metrics 
generated  for  each  target  during  the  comparison  to  the  library  data,  the  weighted  metric,  the  two- 
curve  metric,  and  the  one-curve  metric,  the  feature  space  plots,  and  to  varying  degrees,  the  decay 
thresholds.  Two  Stage  1  classifiers  were  developed  for  the  MMR  data.  One  used  only  the  0.035 
threshold  for  the  time  gates  8-32  decay  to  a  limited  degree;  the  other  used  the  thresholds  for  both 
sets  of  decay  values.  Two  lists  were  used  because  the  0.0045  threshold  for  the  gates  8-43  decay 
appeared  to  remove  a  relatively  large  number  of  targets  plotted  near  some  of  the  library  data. 
Given  the  location  of  these  targets  on  the  space  plot,  it  was  felt  that  keeping  them  on  at  least  one 
list  was  a  conservative  approach  that  needed  to  be  considered  for  this  site.  The  more 
conservative  approach  used  the  following  classifiers: 

•  Priority  -1:  Training  data 

•  Priority  0:  Can’t  analyze 


29 


FINAL 


Figure  6-1:  Feature  Space  Plot  (Decay  8-32)  Used  for  Training  Data  Selection 
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Figure  6-2:  Feature  Space  Plot  (Decay  8-43)  Used  for  Training  Data  Selection 
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•  Priority  1  (Category  1)  -  The  following  are  all  true: 

Weighted  eonfidenee  metric  >  0.575; 

two-curve  confidence  metric  >0.7; 
one-curve  confidence  metric  >0.8;  and 
decay  (gates  8-32)  >  0.035 

•  Priority  2  (Category  1): 

Targets  classified  using  only  two-curve  or  one-curve  matches  and  targets  with  noisy  data 
that  were  relatively  close  to  being  Priority  1  targets. 

•  Priority  3  (Category  2  digs)  -  The  following  are  all  true: 

Weighted  confidence  metric  >  0.8; 

two-curve  confidence  metric  >0.7; 
one-curve  confidence  metric  >0.8;  and 
decay  (gates  8-32)  <  0.035 

•  Priority  4  (Category  2  non-digs)  -  The  following  are  all  true: 

Weighted  confidence  metric  between  0.575  and  0.8; 
two-curve  confidence  metric  >0.7; 

one-curve  confidence  metric  >0.8;  and 
decay  (gates  8-32)  <  0.035 

•  Priority  5  (Category  1): 

Targets  not  meeting  the  above  criteria 

This  approach  did  not  discard  any  relatively  good  matches  to  the  library  data  based  on  gate  8-43 
decay  and  classified  the  better-matching  targets  below  the  gate  8-32  decay  as  “can’t  decide” 
digs.  The  more  aggressive  approach  used  the  following  classifier: 

•  Priority  -1:  Training  data 

•  Priority  0:  Can’t  analyze 

•  Priority  1  (Category  1)  -  The  following  are  all  true: 

Weighted  confidence  metric  >  0.575; 
two-curve  confidence  metric  >0.7; 
one-curve  confidence  metric  >0.8;  and 

decay  (gates  8-32)  >  0.035  or  decay  (gates  8-43)  >  0.0045  or  targets  added  by  the  analyst 
based  on  space  plot  location 

•  Priority  2  (Category  1): 

Targets  classified  using  only  two-curve  or  one-curve  matches  and  targets  with  noisy  data 
that  were  relatively  close  to  being  Priority  1  targets 

•  Priority  3  (Category  2  digs): 

Either  decay  value  below  Priority  1  threshold,  but  space  plot  location  potentially 
indicative  of  TOT  However,  location  or  curve  appearance  not  indicative  enough  of  TOI 
for  immediate  addition  to  Priority  1 .  Cat  2  digs/non-digs  at  discretion  of  analyst. 

•  Priority  4  (Category  2  non-digs)  -  The  following  are  all  true: 

Either  decay  value  below  Priority  1  threshold,  but  space  plot  location  potentially 
indicative  of  TOI.  However,  location  or  curve  appearance  not  indicative  enough  of  TOI 
for  immediate  addition  to  Priority  I .  Cat  2  digs/non-digs  at  discretion  of  analyst. 

•  Priority  5  (Category  1): 

Targets  not  meeting  the  above  criteria 
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The  decision  statistic  used  to  rank  the  targets  for  both  classifiers  was  calculated  as  5  -  priority  + 
weighted  confidence  metric.  The  decision  statistic  was  then  sorted  from  high  to  low  to  order  the 
targets  on  the  dig  lists. 

6.4  DATAPRODUCTS 

6,4,1  Stage  1  Phase  I  Dig  Lists 

The  Stage  1  dig  list  containing  only  the  targets  investigated  during  the  Phase  I  digs  was 
submitted  with  the  following  parameters: 

•  Training  Data:  25  items  selected  as  described  in  Section  6.3.2  (same  for  both  dig  lists). 

•  Can’t  Analyze:  14  targets  that  could  not  be  collected;  the  data  for  all  collected  points 
were  deemed  usable  (same  for  both  dig  lists). 

•  Likely  TOI  (Category  1):  Decision  statistic  greater  than  3.0. 

•  Can’t  Decide  (Category  2  Digs):  Decision  statistic  between  2.0  and  3.0. 

•  Can’t  Decide  (Category  2  Non-Digs):  Decision  statistic  between  1.0  and  2.0. 

•  Likely  Clutter  (Category  3):  Decision  statistic  less  than  1 . 

The  Stage  1  dig  lists  were  only  compared  to  the  project  seed  items  identified  as  “QC  seeds.” 
Due  to  the  differing  strategies  used  for  classification,  the  results  with  respect  to  the  seed  items 
were  different  for  the  two  lists  submitted.  The  results  for  each  are  detailed  below. 

UXAnalyzel 

The  UXAnalyzel  dig  list  was  the  more  aggressive  of  the  two  lists  submitted,  and  five  QC  seeds 
were  missed.  Analysis  of  why  the  seeds  were  missed  was  going  to  be  performed  following 
analysis  of  the  more  conservative  list,  but  the  misses  on  the  more  conservative  list  (discussed 
below)  indicated  that  further  analysis  of  a  more  aggressive  list  would  be  pointless.  Therefore, 
the  analyst  did  not  look  at  which  seeds  were  missed  on  this  list,  simply  noting  that  it  performed 
worse  than  the  more  conservative  list. 

UXAnalvze2 

The  UXAnalyze2  dig  list  was  the  more  conservative  of  the  two  lists  submitted.  Comparison  with 
the  list  of  QC  seeds  indicated  that  three  seeds  would  have  gone  un-dug  based  on  the  Stage  1  list: 
CE-1698  and  CE-1918,  both  intact  81-mm  mortars,  and  CE-2180,  a  155-mm  projectile.  The 
polarization  curves  for  these  three  items  are  shown  in  Eigures  6-3A,  B,  and  C. 
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Figure  6-3:  Polarization  Curves  for  Missed  Quality  Control  Seeds 


A:  CE-1698  (81  mm) 


B:  CE-1918  (81  mm) 


60mm  horizontal,  parallel;  20cm 
wt  met  -  0.6375 


81mm  90  deg,  55cm 
wt  met  -  0.7437 


C:  CE-2180  (155  mm) 


60mm  type  C  (no  tail)  -  vertical 
wt  met  -  0.6346 


As  shown  in  the  figures,  none  of  these  three  items  looks  particularly  ordnance-like  based  on  the 
curves.  CE-1698  and  -1918,  in  particular,  look  more  plate-like  than  cylindrical.  All  three  were 
below  the  0.035  decay  threshold  used  for  this  list,  and  CE-1918  was  the  only  one  of  the  three  that 
had  confidence  metrics  above  all  three  of  the  weighted,  two-curve,  and  one-curve  thresholds.  It 
was  felt  that  modifying  the  classifier  to  identify  these  three  targets  as  items  that  needed  to  be 
intrusively  investigated  would  be  worthless  because  so  many  targets  would  have  to  be  added  as 
digs  that  the  MetalMapper  survey  would  not  end  up  saving  money  over  a  traditional  dig- 
everything  approach.  The  suspected  reason  for  the  non-library-matching  results  was  the  inability 
of  the  multiple-object  solver  used  to  successfully  separate  out  all  of  the  sources  contributing  to 
the  received  signal  for  these  seeds.  The  result  was  two  (or  more)  different  sources  looking  like  a 
single  plate-like  object  rather  than  an  81 -mm  mortar  and  frag.  Because  a  new  version  of 
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UX-Analyze  containing  a  more  advanced  multiple  object  solver  had  been  developed  since  the 
initial  target  inversions  were  performed,  it  was  decided  that  the  new  solver  should  be  tested  on 
the  MMR  dataset.  Revised  results  for  the  three  missed  seeds  are  shown  in  Figures  6-4A,  B,  and 

C. 

While  the  results  don’t  look  significantly  better  for  all  items  —  particularly  CE-1698  and  -2180, 
which  still  look  a  bit  plate-like  —  the  metries,  both  with  respect  to  confidence  and  deeay, 
inereased  to  the  point  that  these  targets  would  now  be  included  as  digs  using  the  former 
classifier.  Therefore,  the  entire  dataset  was  re-inverted  using  the  new  multiple  objeet  solver. 
Rather  than  trying  to  update  the  old  dig  lists  with  new  eonfidence  metrics,  the  classifieation 
process  was  restarted  using  the  re-inverted  data.  The  training  data  and  Can’t  Analyze  datasets 
from  the  UXAnalyzel  and  UXAnalyze2  lists  were  the  used  for  the  new  lists,  except  that  the 
three  missed  QC  seeds  from  the  first  dataset  were  included  as  training  data  for  the  new  lists. 

6,4.2  Stage  1  Dig  Lists  -  Revised 

The  classifieation  rules  for  the  re-inverted  datasets  were  the  same  as  the  original  classification 
rules  with  the  minor  exception  of  the  reduction  of  the  gates  8-43  deeay  threshold  from  0.0045  to 
0.0040  for  the  more  aggressive  of  the  two  lists. 

UXAnalyzeS 

The  more  conservative  elassifier  from  the  original  inversions  was  used  to  eompile  the 
UXAnalyzeS  dig  list.  Beeause  it  was  unknown  whether  any  digs  on  the  first  list  were  classified 
as  likely  clutter  on  this  list,  it  was  compared  to  the  QC  seed  list.  No  QC  seed  items  were  missed. 

UXAnalyzed 

The  more  aggressive  elassifier  from  the  original  inversions  was  used  to  compile  the  UXAnalyze4 
dig  list.  Comparison  with  the  QC  seed  list  indicated  that  one  seed,  CE-2130  (81 -mm  mortar) 
was  incorreetly  identified  as  likely  clutter.  Although  the  confidenee  metrics  for  this  target  were 
well  above  the  thresholds  used  as  cutoffs  for  TOI  (Eigure  6-5),  the  gates  8-43  decay  value  was 
0.0034,  which  was  below  the  threshold  of  0.004.  As  a  result  of  the  miss,  the  threshold  line  for 
the  gates  8-43  decay  value  was  modified  as  shown  in  Eigure  6-6.  Although  targets  with  deeays 
below  the  original  0.004  threshold  were  still  eonsidered  non-digs  to  begin  with,  additional 
eonsideration  was  given  to  adding  any  that  looked  like  partieularly  strong  matches  to  TOI, 
including  CE-2130. 
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Figure  6-4:  Revised  Polarization  Curves  for  Missed  Quality  Control  Seeds 


A:  CE-1698  (81  mm) 


B:  CE-1918  (81  mm) 


81mm  mortar  -  fully  intact,  20in;  horiz;  35cm  (MMR) 
wt  met  -  0.6456 


155mm  projectile  -  solid,  24in;  nose  dn;  60cm  (MMR) 
wt  met  -  0.6198 


Time  (ml) 


C:  CE-2180  (155  mm) 


Medium  iSO;  horiz;  30cm  (MMR) 
wt  met  -  0.6593 
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Figure  6-5:  Polarization  Curves  and  Confidence  Metrics  for  CE-2130 

CE-2130m1 


105mm  HEAT  -  fully  Intact,  26ln;  horiz;  50cm  (MMR) 
wt  met  -  0.9073 


81mm  mortar  -  fully  Intact,  20ln;  horiz;  35cm  (MMR) 
wt  met  -  0.8256 


105mm  HEAT  -  fully  Intact,  26ln;  horiz;  50cm  (MMR) 
crv2  met  -  0.9093 


105mm  HEAT  -  fully  Intact,  26ln;  nose  up;  50cm  (MMR) 
crv2  met  -  0.8937 


81mm  horizontal,  parallel;  25cm 
crvi  met  -  0.9807 


Time  (n^) 

105mm  HEAT  -  fully  intact,  26in;  45  deg;  50cm  (MMR) 
crvi  met  -  0.9533 


Top  row  =  weighted  metric;  middle  =  2-curve,  bottom  =  1-curve 
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Figure  6-6:  Missed  Quality  Control  Seed  in  Feature  Space 


MMR  -  Decay  (8-43)  vs  Size  -  P1 
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6.4.3  Dig  List  Stage  2 

Based  on  the  few  misses  for  either  Stage  1  dig  list,  ground  truth,  typieally  delivered  after  the 
submission  of  Stage  2  lists,  was  delivered  based  on  the  targets  marked  as  digs  on  the  Stage  1  dig 
lists.  No  major  modifieations  were  made  to  either  of  the  elassifiers  used  for  the  projeet.  Nine 
targets  were  changed  from  non-digs  to  digs  on  the  UXAnalyzeS  list  based  on  their  proximity  in 
feature  space  to  items  that  were  revealed  to  be  TOI  in  the  ground  truth.  All  of  these  targets  had 
been  screened  out  based  on  their  gate  8-32  decay  values  in  Stage  1 .  Seven  digs  were  added  to 
the  UXAnalyzed  list  based  either  on  their  proximity  to  ground  truth  TOI  in  the  feature  space  or 
on  the  revision  of  the  gates  8-43  decay  threshold  shown  in  Figure  6-6.  Ground  truth  for  the  new 
digs  was  returned  following  the  submission  of  the  Stage  2  lists.  None  of  the  added  digs  on  either 
list  were  TOI. 

6.4.4  Dig  List  Stage  3 

None  of  the  targets  added  to  either  dig  list  following  Stage  1  were  TOI,  so  there  was  no  reason  to 
modify  the  classifiers  or  add  any  additional  targets  for  Stage  3.  Therefore,  both  lists  were 
compared  to  the  full  Phase  I  ground  truth  results  without  any  changes  to  the  Stage  2  lists.  The 
final  UXAnalyze3  dig  list  included  373  digs  out  of  a  total  of  895  Phase  I  targets  (42%  dig  rate), 
and  the  UXAnalyze4  dig  list  included  309  digs  (35%  dig  rate). 

6.4.5  Phase  II  Dig  List 

The  results  of  the  comparison  of  the  two  Phase  I  dig  lists  were  used  to  determine  the  classifier  to 
be  used  to  rank  the  unexcavated  targets  remaining  at  the  site  following  the  Phase  I  intrusive 
operation.  The  results,  detailed  in  Section  7,  indicated  that  the  TOI  detected  versus  non-TOI 
removed  from  consideration  as  dig  targets  statistics  were  better  for  the  UXAnalyze4  classifier. 
Therefore,  this  classifier  was  used  to  compile  the  dig  list  for  the  Phase  II  targets. 
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7.0  PERFORMANCE  ASSESSMENT 

7.1  OBJECTIVE:  REPEATABILITY  OF  INSTRUMENT  VERIFICATION  STRIP 
MEASUREMENTS 

One  IVS  seed  item  was  ineorreetly  elassified  (i.e.,  weighted  eonfidenee  that  the  seed  item  was 
the  seed  item  or  a  similarly-sized  TOI  was  less  than  0.7)  during  the  projeet.  Beeause  this  failure 
was  out  of  a  dataset  of  105  IVS  items  surveyed  over  the  eourse  of  the  projeet,  the  IVS  failure  rate 
was  slightly  less  than  1%.  The  ineorrect  elassification  oecurred  during  the  afternoon  survey  of 
the  IVS  on  June  19,  2012.  It  is  apparent  that  the  data  are  poor  given  the  laek  of  a  third 
polarizability  curve.  Because  poor  data  are  collected  occasionally  with  the  MetalMapper,  the 
“Can’t  Analyze”  classification  has  been  used  to  identify  targets  that  must  be  dug  given  unusable 
MetalMapper  data.  Had  this  target  been  in  the  field  rather  than  in  the  IVS,  it  would  have  been 
classified  as  a  Can’t  Analyze  and  would  have  been  dug. 

The  standard  deviations  of  the  horizontal  and  vertical  differences  in  location  between  the  actual 
and  modeled  locations  of  the  seed  items  were  4.1  cm  and  4.7  cm,  respectively.  Both  are  well 
within  the  stated  goals  of  15  cm  for  the  horizontal  and  10  cm  for  the  vertical. 

7.2  OBJECTIVE:  CORRECTLY  POSITION  METALMAPPER  RELATIVE  TO 
SOURCE 

The  results  used  for  209  of  the  895  Phase  I  targets  had  a  modeled  location  greater  than  40  cm 
from  the  collection  location  but  were  judged  usable  by  the  analyst  based  on  three  factors: 

•  The  target  was  classified  as  a  TOI  despite  having  a  modeled  location  greater  than  40  cm 
from  the  collection  location; 

•  The  operator  noted  during  collection  that  there  was  no  source  evident  at  the  dynamic 
target  location  (based  on  the  dancing  arrows  display),  so  data  was  collected  directly  on 
top  of  the  dynamic  location  to  satisfy  the  requirements  of  the  collection  to  dynamic  target 
distance  data  quality  objective;  or 

•  The  version  of  the  target  selected  for  classification  (multiple  object  solver  version  of  the 
point  selected  rather  than  single  object  solver  version)  was  more  indicative  of  TOI  than  a 
modeled  point  within  40  cm  of  the  collection  location  despite  being  outside  the  data 
quality  objective  radius. 

Two  hundred  thirteen  of  the  1,392  Phase  II  targets  had  a  modeled  location  greater  than  40  cm 
from  the  collection  location  for  the  same  reasons. 

7.3  CORRECTLY  POSITION  METALMAPPER  RELATIVE  TO  EM61-MK2  TARGET 

One  MetalMapper  collection  point  was  within  73  cm  of  every  EM61-MK2  target  location. 
Seventeen  of  the  selected  results  were  farther  than  73  cm  from  their  dynamic  selection  locations 
but  were  judged  usable  by  the  analyst  based  on  the  following  factors: 

•  The  target  was  classified  as  a  TOI  despite  the  offset  between  the  collection  location  and 
the  dynamic  target; 

•  The  collected  data  looked  reasonable  (i.e.,  inversion  good,  no  noise)  and  there  were  no 
other  dynamic  targets  within  1 .5  meters  of  the  dynamic  target  in  question,  signifying  that 
the  operator  collected  data  over  the  intended  source;  or 
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•  The  collection  location  was  outside  the  dynamic  survey  area,  indicative  of  a  source  that 
was  not  fully  covered  by  the  dynamic  survey. 

7.4  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  TARGETS  OF 
INTEREST 


7,4,1  Phase  I  Results 

The  submitted  Stage  3  dig  lists  were  compared  to  the  complete  set  of  the  Phase  I  ground  truth 
data  from  MMR  by  the  IDA.  This  comparison  identified  the  same  five  TOI  that  would  have 
gone  un-dug  on  both  lists: 

•  CE- 1381:  60-mm  mortar 

•  CE-2089:  parachute  flare 

•  CE-2128:  cartridge 

•  CE-2141:  155-mm  projectile 

•  CE-2 167:  3 7-mm  projectile 

Out  of  the  895  targets  classified  for  Phase  I  of  the  MMR  project,  148  were  considered  TOI 
following  the  intrusive  investigation.  Given  the  five  misses,  143  of  the  148  TOI  (96.6%)  at  the 
site  were  correctly  classified  as  targets  that  should  be  dug.  Eigures  7-1  and  7-2  show  the  receiver 
operating  characteristic  (ROC)  curve  for  the  two  MMR  dig  lists. 


Figure  7-1:  Receiver  Operating  Characteristic  Curve  for  UXAnalyzeS  Dig  List 


MM 


MMR1  Parsons  UXAnalyzeS  None  MetalMapper  Custom  s2  v1 


Difficult  TOIs: 

#CE-1381  (60mm  legacy) 
#CE-2128  (Cartridge  legacy) 
#CE-2092  (ParachuteFlare  legacy) 
#CE-2167  (37mm  legacy) 
#CE-2141  (155mm  legacy) 
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Figure  7-2:  Receiver  Operating  Characteristic  Curve  for  UXAnalyze4  Dig  List 


MM 

MMR1  Parsons  UXAnalyze4  None  MetalMapper  Custom  s2  v1 


Difficult  TOis: 

#CE-1381  (60mm  legacy) 
#CE-2128  (Cartridge  legacy) 
#CE-2167  (37mm  legacy) 
#CE-2089  (ParachuteFlare  legacy) 
#CE-2141  (155mm  legacy) 


Although  identified  as  TOI  by  the  IDA  during  seoring,  three  of  the  five  missed  items  were  not 
eonsidered  high-priority  removals  given  the  goals  of  the  project  (i.e.,  the  identification/removal 
of  large  munitions  containing  the  majority  of  the  energetic  filler  that  could  contribute  to 
groundwater  contamination).  Only  the  60-mm  mortar  (CE-1381)  and  the  155-mm  projectile 
(CE-2141)  were  identified  as  munitions  of  primary  interest  in  the  ESTCP  Demonstration  Plan 
(ESTCP,  2012).  CE-1381  was  not  identified  as  a  TOI  based  on  the  two-curve  and  one-curve 
confidence  metrics,  which  were  both  below  their  respective  thresholds  of  0.700  and  0.800 
(Eigure  7-3). 

It  is  suspected  that  the  60-mm  mortar  examples  in  the  library  did  not  match  the  recovered 
illumination  round  warhead  very  well.  There  were  four  versions  of  60-mm  mortar  rounds  in  the 
library,  three  of  which  were  in  the  standard  MetalMapper  library  supplied  with  UX-Analyze  and 
one  of  which  was  from  test  pit  data  collected  at  the  former  Camp  Beale  in  California.  Because 
there  are  no  pictures  of  the  three  versions  in  the  default  MetalMapper  library,  it  is  unknown  how 
closely  they  match  the  one  recovered  at  MMR.  However,  Figure  7-4  illustrates  the  differences 
between  the  MMR  and  Camp  Beale  mortars. 
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Figure  7-3:  Polarization  Curves  and  Confidence  Metrics  for  CE-1381 


CE-1381 


60tnm  type  A  -  vertical,  nose  down 
wt_met  •  0.6545 


60mm  type  A  •  vertical,  nose  down 
Cfv2_met  -  0  6538 


81mm  mortar*  fully  intact.  20in;  nose  dn:  35cm  (MMR) 
crvi  met  -  0.6951 


81mm  type  B  -  vertical,  nose  up 
wt_met  •  0.6443 


81mm  type  B  -  inclined,  nose  down 
Cfv2_met  -  0.6491 


105mm  heat  type  A  -  vertical,  nose  down 
crvi  met -0.5494 


Top  row  =  weighted  metric;  middle  =  two-curve,  bottom  =  one-curve 
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Figure  7-4:  60-mm  Mortar  Types 
A:  MMR  B:  Beale  Test  Pit 


Anomalv  Date  08-01-12 

Depth- 

LeNCFTH-  22Cm  Dxq  X/p£  mo 


Because  there  were  possibly  significant  differences  between  the  60-mm  examples  in  the  library 
and  the  recovered  item,  a  possible  solution  to  the  false  negative  was  adding  this  item  to  the 
library.  However,  the  two  decay  values  calculated  for  this  item  (0.019  for  time  gates  8-32  and 
0.0007  for  gates  8-43)  were  low  enough  that  it  would  have  been  classified  as  non-TOI  even  if 
the  confidence  metrics  had  been  higher.  Adding  this  item  to  the  library  and  modifying  the  decay 
thresholds  to  select  it  as  a  target  would  have  added  a  substantial  number  of  digs  to  the  project. 
Given  the  project  goal  of  reducing  the  amount  of  energetic  material  in  the  subsurface  rather  than 
removing  every  piece  of  TOI,  it  was  not  recommended  that  these  changes  be  made  to  identify 
one  specific  partial  60-mm  mortar  round. 

The  ground  truth  results  for  CE-2141,  the  missed  155-mm  projectile,  did  not  include  a  recovered 
location.  Therefore,  it  is  difficult  to  make  any  definitive  statements  regarding  the 
misclassification  of  this  item  as  non-TOI.  The  multiple-object  inversion  returned  two  TOI-like 
results  for  this  item,  one  that  matched  an  81 -mm  mortar  with  weighted,  two-curve,  one-curve 
confidence  metrics  of  0.73,  0.73,  and  0.90,  and  one  that  matched  a  60-mm  mortar  with 
confidence  metrics  of  0.82,  0.81,  and  0.90.  Based  on  the  higher  confidence  metrics  and  an 
extremely  low  gate  8-43  decay  value  of  0.0009  for  the  source  that  matched  the  81 -mm  mortar, 
the  source  matching  the  60-mm  mortar  was  selected  as  the  result  for  this  target.  It  was  ultimately 
not  selected  as  a  dig  based  on  its  decay  values  (0.020  for  time  gates  8-32  and  0.0023  for  gates  8- 
43)  rather  than  confidence  metrics. 

However,  because  there  was  no  recovery  location,  it  is  unclear  exactly  how  close  the  recovered 
155-mm  projectile  was  to  either  of  the  modeled  results.  In  addition,  the  result  for  CE-2151 
suggested  that  this  target  was  a  105-mm  HEAT  round  that  should  be  dug.  The  modeled  location 
of  this  target  was  approximately  1.2  meters  from  the  recommended  dig  location  for  CE-2141. 
CE-2151  was  not  intrusively  investigated  during  the  Phase  I  or  Phase  II  intrusive  operations, 
apparently  due  to  a  lack  of  time.  It  is  possible  that  the  155-mm  may  actually  have  been 
recovered  closer  to  that  EM61-MK2  target  and  been  the  reason  for  the  “dig”  recommendation 
there.  Without  more  detail  about  where  the  155-mm  listed  as  the  results  for  CE-2141  was 
actually  recovered,  it  was  not  recommended  that  the  classifier  be  modified  to  detect  it. 
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7.4,2  Phase  II  Results 

The  single  dig  list  submitted  for  Phase  II  of  excavation  was  compared  to  the  complete  set  of 
Phase  II  ground  truth  data  by  Parsons.  Of  the  1,386  targets  remaining  uninvestigated  following 
Phase  I,  only  1,049  were  excavated  during  Phase  II  due  to  time  constraints  at  the  site. 

Twenty  of  115  items  identified  as  TOI  following  the  intrusive  investigation  were  incorrectly 
classified  as  non-TOI  in  the  Phase  II  dig  list,  meaning  that  only  83%  of  the  TOI  were  correctly 
classified  as  TOI.  One  of  the  false  negatives  was  for  a  37-mm  projectile,  which  was  not 
considered  a  high-priority  target,  as  discussed  in  Section  7.4.1.  The  other  19  were  60-mm 
projectiles,  17  of  which  were  the  same  type  of  illumination  round  warhead  shown  in  Figure  7- 
4A.  The  other  two  were  fuzed  HE  rounds,  which  were  again  thought  to  be  dissimilar  to  the 
60-mm  mortar  examples  in  the  classification  library.  Because  so  many  more  illumination  rounds 
were  recovered  during  the  Phase  II  intrusive  investigation,  three  examples  of  these  and  one  of  the 
HE  rounds  were  added  to  the  classification  library,  and  the  full  dataset  was  re-compared  to  the 
library.  Addition  of  the  illumination  round  examples  to  the  library  resulted  in  the  detection  of  all 
of  the  others  using  the  current  classifier  (without  any  decay  thresholds).  However,  the  number  of 
digs  added  to  correctly  identify  these  as  TOI  appears  to  be  huge  (more  than  700  upon  a  quick 
review).  A  more  thorough  reclassification  effort  would  need  to  be  performed  to  determine  the 
exact  impact  on  the  dig  rate  at  the  site.  However,  it  appears  these  warheads  are  quite  similar 
geophysically  (smaller  size  and  faster  decay  than  most  of  the  other  TOI  identified  at  the  site  and 
polarizability  curves  similar  to  many  of  the  fragments  or  fragment  pits)  to  a  great  deal  of  the 
clutter  at  the  site. 

7.5  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  NON-TARGETS  OF 
INTEREST 

7.5,1  Phase  I  Results 

The  submitted  UXAnalyze3  dig  list  correctly  identified  69%  (517  of  747)  of  the  clutter  as  clutter 
in  the  dataset,  although  this  did  include  the  misidentification  of  five  TOI  as  non-TOI.  The 
UXAnalyze4  dig  list  correctly  identified  78%  (581  of  747)  of  the  clutter  as  clutter  with  the  same 
five  missed  TOI.  As  discussed  in  Section  7.4,  no  modifications  were  recommended  for  either 
classifier  in  order  to  consider  these  TOI  “detected.”  With  no  modifications,  these  percentages 
could  be  considered  the  approximate  reductions  in  clutter  digs  that  would  be  seen  using  either  of 
the  classifiers. 

However,  as  indicated  in  Figures  7-5  and  7-6,  there  were  no  TOI  recovered  at  the  site  with  a  size 
less  than  approximately  0.8.  It  therefore  seems  reasonable  to  set  a  bottom  threshold  for  the  size 
of  a  target  to  be  considered  TOI,  as  has  been  done  with  the  two  decay  values.  In  the  case  of  the 
Phase  I  targets,  the  removal  of  any  targets  with  sizes  less  than  0.65  would  have  reduced  the 
number  of  dig  targets  on  the  ElXAnalyze3  dig  list  by  44  (75%  reduction  in  clutter  digs)  and  those 
on  the  UXAnalyze4  list  by  30  (82%  reduction  in  clutter  digs).  The  Phase  II  dig  list  for  MMR 
was  submitted  using  this  threshold.  Future  use  would  likely  depend  on  the  results  of  the  Phase  II 
dig  list  versus  seed  item  comparison. 

The  dashed  lines  on  both  figures  are  the  initial  decay  thresholds  used  to  separate  TOI  from  likely 
clutter.  Re-addition  as  potential  TOI  was  strongly  considered  for  any  targets  above/to  the  right 
of  the  solid  lines,  with  re-addition  based  on  confidence  metrics  and  polarization  curve 
appearance.  The  TOI  below  the  solid  lines  on  the  figures  were  the  five  missed  items  discussed  in 
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Section  7.4,  were  correctly  classified  as  targets  that  should  be  dug  based  on  the  use  of  less  than 
three  polarization  curves  (somewhat  poor  data),  or  were  sources  for  which  more  than  one  point 
was  collected  (i.e.,  two  or  more  EM61-MK2  targets  were  picked  on  the  same  source).  In  the 
case  of  the  sources  with  more  than  one  EM61-MK2  target,  at  least  one  of  the  MetalMapper 
points  collected  in  the  vicinity  was  labeled  as  a  dig. 

7,5.2  Phase  II  Results 

Prior  to  the  re-comparison  of  the  full  dataset  to  the  classification  library  containing  the  60-mm 
illumination  round  warheads  and  the  HE  mortar  example,  the  Phase  II  dig  list  contained  407  digs 
for  1,392  targets,  including  the  115  TOI  in  both  numbers.  Without  the  TOI,  there  were  292  non- 
TOI  digs  for  1,277  non-TOI  targets,  which  represented  a  reduction  in  non-TOI  digs  of 
approximately  77%.  As  mentioned  in  Section  7.4.2,  a  quick  re-classification  effort  indicated  that 
adding  the  illumination  rounds  to  the  library  and  disregarding  the  decay  thresholds  currently 
used  in  the  classifier  would  significantly  decrease  the  amount  of  clutter  that  could  be  left  in  the 
ground  at  this  site. 

7.6  OBJECTIVE:  CORRECT  SPECIFICATION  OF  NO-DIG  THRESHOLD 

7.6.1  Phase  I  Results 

Because  items  were  missed  in  the  comparison  of  the  dig  lists  to  the  ground  truth  set,  the  no-dig 
thresholds  on  the  submitted  dig  lists  were  set  incorrectly.  Target  CE-1381,  the  missed  60-mm 
mortar,  was  339  spots  below  the  dig  threshold  in  the  UXAnalyze3  dig  list  and  396  spots  below 
the  threshold  on  the  UXAnalyze4  dig  list.  As  indicated  in  Section  7.4,  it  seemed  that  modifying 
the  classifiers  to  detect  this  item  would  result  in  so  many  additional  digs  that  the  use  of  the 
MetalMapper  at  MMR  would  not  be  particularly  worthwhile.  Given  the  nature  of  the  problem  at 
MMR  (groundwater  contamination  from  energetic  material  versus  explosive  hazard),  the  stop  dig 
thresholds  were  deemed  appropriate  for  both  dig  lists. 

7.6.2  Phase  II  Results 

The  recovery  of  17  of  the  60-mm  illumination  round  warheads  during  the  Phase  II  intrusive 
investigation  indicated  that  they  were  much  more  prevalent  than  indicated  by  the  Phase  I 
investigation.  As  with  the  Phase  I  list,  the  illumination  rounds  were  relatively  low  priority 
within  the  no-dig  section  of  the  ranked  dig  list.  A  retrospective  analysis  of  the  data  performed 
after  adding  examples  of  the  illumination  rounds  to  the  classification  library  indicated  that  they 
could  be  identified  as  TOI  using  a  classifier  similar  to  the  one  used  to  compile  the  Phase  I  and 
Phase  II  dig  lists.  However,  the  number  of  dig  targets  added  to  the  dig  list  in  order  to  find  these 
warheads  appears  to  be  quite  significant  if  only  the  confidence  metrics  used  in  the  classifier  are 
considered.  Additional  effort  would  be  necessary  to  perform  a  full  reclassification  of  the  data 
using  the  modified  library  and  to  determine  if  additional  filters  could  be  developed  using 
size/decay  values  or  other  geophysical  characteristics  to  separate  the  items  from  much  of  the 
clutter  at  the  site. 
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Figure  7-5:  Feature  Space  Plot  (Decay  8-32)  Showing  Targets  of  Interest  Recovered  during 

Massachusetts  Military  Reservation  Phase  I 
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Figure  7-6:  Feature  Space  Plot  (Decay  8-43)  Showing  Targets  of  Interest  Recovered  during 

Massachusetts  Military  Reservation  Phase  I 
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7.7  MINIMIZE  NUMBER  OF  ANOMALIES  THAT  CANNOT  BE  ANALYZED 

The  same  14  targets  were  classified  as  Can’t  Analyze  on  the  dig  lists  submitted.  All  14  were 
targets  that  could  not  be  collected  due  to  topography  and/or  vegetation.  All  collected  points  were 
deemed  usable.  Because  topography  is  beyond  the  scope  of  the  field  team  or  data  analyst  to 
control,  the  Can’t  Analyze  rate  for  this  project  is  considered  to  be  0%. 

7.8  CORRECT  ESTIMATION  OF  TARGET  PARAMETERS 

The  target  parameters  estimated  in  this  case  were  the  X,  Y,  and  relative  Z  (depth)  coordinates  of 
the  targets.  Because  the  goal  with  this  objective  is  to  direct  the  dig  teams  to  the  correct  locations 
for  TOI,  the  comparison  of  estimated  coordinates  to  actual  coordinates  was  performed  only  for 
TOl  and  for  those  targets  marked  as  digs  in  the  ranked  dig  list. 

7.8.1  Phase  I  Results 

The  success  criteria  for  this  performance  objective  were  X,  Y  offsets  for  which  one  standard 
deviation  of  the  dataset  was  less  than  15  cm  and  one  standard  deviation  of  the  depth  offset  was 
less  than  10  cm.  The  horizontal  offset  for  the  TOl  dataset  was  above  the  performance  objective, 
with  a  calculated  standard  deviation  of  28  cm,  as  was  the  depth  offset,  with  a  calculated  standard 
deviation  of  18  cm.  The  results  for  all  targets  marked  as  digs  were  similar,  with  calculated 
standard  deviations  of  30  cm  for  the  horizontal  offset  and  14  cm  for  the  vertical  offset.  It  is 
assumed  that  a  combination  of  the  high  target  densities  at  the  site  and  the  relatively  large  size  of 
the  objects  being  recovered  are  responsible  for  the  larger-than-expected  horizontal  offsets.  The 
high  target  densities  could  be  creating  problems  for  the  multiple  object  solver,  which  may  not  be 
modeling  location  as  effectively  as  anticipated.  The  intrusive  team  also  used  the  same  central 
point  as  the  defined  location  of  the  “source”  if  multiple  objects  were  recovered  near  one  another. 
Many  of  the  objects  recovered  during  this  project  were  quite  large  compared  to  those  recovered 
at  other  sites,  so  there  may  be  some  discrepancy  between  the  center  of  mass  per  the  intrusive 
team  and  the  center  of  mass  per  the  modeled  results,  particularly  in  the  case  of  locations  with 
multiple  objects  recovered.  Finally,  the  site  was  covered  in  craters,  stumps,  and  brush  piles.  A 
great  deal  of  the  vertical  offset,  which  is  quite  a  bit  higher  than  that  seen  on  previous 
demonstration  sites,  may  be  because  the  data  collection  team  was  often  unable  to  lay  the 
MetalMapper  flush  with  the  ground  surface. 

7.8.2  Phase  II  Results 

The  standard  deviation  for  the  Phase  II  TOI  horizontal  offset  was  19  cm,  and  the  standard 
deviation  for  the  Phase  II  vertical  offset  was  11  cm.  Both  were  above  the  performance 
objectives,  although  they  were  much  closer  to  the  objectives  than  the  Phase  I  data  were.  It  is 
expected  that  a  greater  emphasis  put  on  recording  GPS  coordinates  for  each  recovered  item 
during  Phase  II  was  the  reason  for  the  increased  accuracy.  The  horizontal  and  vertical  offsets  for 
the  dig  targets  were  both  15  cm  for  the  Phase  II  intrusive  effort. 
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8.0  COST  ASSESSMENT 

The  cost  assessment  was  split  into  two  groups:  MetalMapper  costs  and  conventional  intrusive 
costs.  The  MetalMapper  costs  include  instruments,  surveying,  seeding,  and  analysis  costs;  the 
conventional  intrusive  costs  include  surface  sweep,  data  collection,  and  intrusive  costs. 

8.1  COST  MODEL 

The  cost  model  for  the  MMR  demonstration  includes  the  total  cost  of  the  project  and  potential 
savings  from  the  classification  process.  The  total  cost  includes  the  seeding  operation, 
MetalMapper  operations,  processing,  and  intrusive  operation.  Estimates  for  each  operation  are 
listed  in  Table  8-1. 
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Table  8-1:  Details  of  Costs  Tracked 


Cost  Element 

Data  Tracked  During  Demonstration 

Estimated  Costs 

Seeding  Costs 

Seed  emplacement  / 

Costs  for  surface  sweep,  seed  emplacement, 

$45,103 

initial  set  up 

surveying  seeds 

MetalMapper  Survey  Costs 

MetalMapper  rental  ($500/day;  29  days) 

$14,500 

MetalMapper  prep  fee  (project) 

$1,000 

MetalMapper  shipping  (project) 

$2,769 

Tractor  rental  (project) 

$1,365 

Tractor  mob/demob  (project) 

$250 

Instrument  Costs 

$2,400 

RTK  GPS  cost  ($800/week;  3  weeks) 

Shipping  (RTK  GPS,  etc;  project) 

$324 

Total 

$22,608 

Per  target 

$9.88 

Field-related  labor  (two  geophysicists), 
equipment  setup,  test  pit  data  collection, 
cued  data  collection,  preprocessing,  initial 

$46,995 

Survey  Costs 

target  inversion  for  quality  control  checks, 
non-equipment  direct  costs  (e.g.,  per  diem, 
hotel,  truck  rental,  fuel) 

Per  target 

$20.54 

All  processing  and  analysis  performed 

$8,626 

Analysis  Costs 

following  the  completion  of  field  activities 

Per  target 

$3.77 

Intrusive  Costs 

All  costs  related  to  the  intrusive 

$363,455 

Investigations 

investigation 

Cost  per  anomaly  to  intrusively  investigate 

$152 
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Although  the  MetalMapper  used  for  data  colleetion  at  MMR  was  funded  by  ESTCP,  standard 
rental  costs  and  prep  fees  were  used  to  determine  the  cost  for  the  MMR  project  had  Parsons 
rented  the  MetalMapper  for  the  duration  of  the  project.  Survey  and  analysis  costs  were  tracked 
using  a  task-specific  number  in  Parsons’  project  controls  system.  Although  production  rates 
were  higher  on  this  project  than  on  previous  demonstration  projects,  another  reason  for  the 
significant  drop  in  MetalMapper  survey  costs  was  the  addition  of  a  UXO  technician  to  the  field 
team  for  safety  purposes  on  Parsons’  two  earlier  field  projects.  Given  the  full  surface  sweep 
performed  before  this  project,  the  tech  was  unnecessary  at  MMR. 

The  final  result  of  the  classification  process  provides  an  alternative  approach  to  the  final  costs  of 
the  intrusive  operations.  The  analysis  compares  costs  of  using  MetalMapper  versus  digging  all 
anomalies  classified  as  no-digs.  The  cost  model  assumes  the  use  of  the  UXAnalyzed  classifier. 
As  discussed  in  Section  7.4,  this  classifier  is  expected  to  reduce  the  number  of  clutter  digs 
necessary  at  the  she  by  as  much  as  82%,  although  the  proven  result  of  78%  for  Phase  I  will  be 
used  for  the  model.  The  cost  of  MetalMapper  classification  and  analysis  at  MMR  was  $34.19 
per  target. 

The  overall  cost  of  excavating  the  2,381  Phase  I  and  Phase  II  anomalies  was  approximately 
$363,455  ($152/anomaly).  However,  this  is  a  factor  of  the  relatively  small  scale  of  the 
demonstration  project  and  the  challenging  vegetation  conditions  at  this  site.  As  shown  in 
Section  8.3,  the  MetalMapper  would  likely  prove  to  be  a  cost-saving  tool  on  a  larger  scale 
project. 

8.2  COST  DRIVERS 

Based  on  the  factors  described  above,  the  total  per-target  cost  for  the  MetalMapper-related  work 
of  the  MMR  project  was  $34.19.  The  production  rate  for  the  project  was  207  targets  collected 
per  day,  which  did  not  include  re-shots  as  part  of  the  collection.  The  average  production  rate 
prior  to  the  collection  of  re-shots  was  227  targets  per  day.  These  seem  like  achievable  rates  at 
this  she;  therefore.  Parsons  considers  $34.19  per  target  a  reasonable  estimation  of  the  costs  that 
would  be  expected  for  future  projects  at  MMR. 

For  the  intrusive  investigation  phase  at  MMR,  Parsons  investigated  2,381  anomalies  in  36 
intrusive  days.  The  average  anomalies  intrusively  investigated  per  day  was  68  anomalies,  which 
was  fewer  than  the  estimated  80  anomalies  per  day  planned  for  the  project.  The  decrease  in 
productivity  was  almost  entirely  due  to  the  need  to  clear  multiple  sources  at  each  target  location. 
The  most  common  item  excavated  at  MMR  was  MD  (frag). 

8.3  COST-BENEFIT 

For  a  production  removal  action  project  with  10,000  anomalies  selected  for  investigation,  the 
$34.19/anomaly  cost  calculated  for  the  demonstration  was  used  for  data  collection  and 
processing.  The  intrusive  costs  is  expected  to  be  closer  to  $100/anomaly.  Assuming  a  60% 
reduction  in  the  number  of  clutter  items  that  could  be  eliminated  from  intrusive  investigation 
would  yield  a  potential  cost  savings  based  on  the  following  assumptions: 

•  10,000  anomalies  at  $100/anomaly  for  intrusive  investigation  equals  a  cost  of 
$1,000,000. 

•  Reduction  of  6,000  anomalies  equals  a  reduction  of  $600,000  in  excavation  costs. 
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•  MetalMapper  costs  for  collecting  and  analyzing  10,000  anomalies  at  $34.19/anomaly 
equals  $341,900. 

•  Total  net  savings  under  this  seenario  equals  $258,100  (26%). 
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9.0  IMPLEMENTATION  ISSUES 

There  were  a  few  notable  implementation  issues  regarding  the  MMR  projeet: 

•  The  greatest  ehallenge  for  the  intrusive  portion  of  the  projeet  was  the  sheer  amount  of 
elutter  present  at  the  site.  While  only  relatively  large  dynamic  targets  (with  regard  to 
instrument  response)  were  selected  for  follow-on  investigation  with  the  advanced  EMI 
sensors,  most  of  the  site  was  littered  with  smaller  fragments  that  were  not  picked  as  cued 
targets.  These  fragments  made  clearing  intrusive  locations  particularly  difficult  and 
required  quite  a  bit  more  time  to  locate  and  catalog  than  on  previous  projects. 

•  The  MetalMapper  data  collection  went  very  well  for  this  project.  There  were  few 
software  or  hardware  issues  except  for  a  broken  inertial  measuring  unit  (IMU)  cable. 
Because  the  IMU  data  were  not  essential  to  the  collection  or  processing  of  the 
MetalMapper  data,  the  field  team  carried  on  without  it  for  about  2  days  during  collection. 
It  was  thought  that  the  data  collected  without  a  functioning  IMU  might  show  greater 
offsets  to  the  recovered  sources  than  that  collected  with  the  IMU.  However,  for  the  I  day 
without  the  IMU  with  dig  results  (June  21,  2012;  points  collected  during  the  partial  days 
without  the  IMU  were  not  investigated  during  the  Phase  I  digging)  showed  little  offset 
difference  from  the  rest  of  the  data.  The  standard  deviation  of  offsets  for  June  21  was 
29  cm  versus  28  cm  for  all  of  the  Phase  I  results. 

•  The  only  other  MetalMapper  implementation  issues  in  the  field  were  due  to  the  rough 
condition  of  the  site  (e.g.,  craters,  stumps,  fallen  trees,  brush)  leading  to  slow  drive  times 
to  the  farthest  end  of  the  site  and  the  lack  of  available  background  point  collection 
locations.  The  data  seem  to  be  of  sufficient  quality  for  use  in  discriminating  large  TOI 
from  the  vast  amounts  of  clutter  at  the  site,  but  it  is  unknown  how  different  geophysically 
the  background  location  at  the  IVS  strip  was  from  the  field  site. 

•  The  misclassification  of  some  60-mm  illumination  round  warheads  as  clutter  was  the 
largest  issue  with  the  classification  portion  of  the  project.  These  rounds  appear  to  be 
much  more  geophysically  similar  to  much  of  the  clutter  at  the  site  (relatively  small  size 
and  fast  decay)  than  they  are  to  the  larger  TOI  present  and  even  to  thicker-walled  60-mm 
mortars.  It  appears  that  attempting  to  consistently  classify  them  as  TOI  would  add  a 
significant  number  of  clutter  digs  to  any  intrusive  investigation.  It  may  be  worth 
considering  their  potential  impact  at  the  site  with  regard  to  the  overriding  groundwater 
concerns  to  determine  whether  adding  digs  to  recover  them  is  worthwhile. 
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